Help needed to tune hoverboard motor with encoder

I have a pair of hoverboard motors and AMT102-V CUI encoders attached to Odrive 3.4 and bench power supply up to 24V but I set it to 13V. Pulley ratio 3:1.
I calibrated the motor and encoder and during startup this seems to be doing correctly.
But when I wan to to change to CLOSED LOOP state, then motor starts oscillating, moving back an d forth very quickly without setting any velocity setpoint.
I suppose that the issue is with tuning parameters, P, I, D, and current limits or others, but I don’t know how to start.
I checked the encoder mounting and shaft, the vel_gain and pos_gain values seems low enough for me.
When I raise the motor curren_lim from 10 to 15, then no immediate oscillating, but setting any vel_setpoint leads to short quick move of motor and after that I lose communication with Odrive.

error = 0x0000 (int)
loop_counter = 207803 (int)
current_state = 1 (int)
config:
  startup_motor_calibration = True (bool)
  spin_up_acceleration = 400.0 (float)
  startup_sensorless_control = False (bool)
  step_gpio_pin = 1 (int)
  spin_up_current = 10.0 (float)
  startup_encoder_index_search = True (bool)
  ramp_up_time = 0.4000000059604645 (float)
  counts_per_step = 2.0 (float)
  dir_gpio_pin = 2 (int)
  spin_up_target_vel = 400.0 (float)
  startup_closed_loop_control = False (bool)
  startup_encoder_offset_calibration = True (bool)
  ramp_up_distance = 12.566370964050293 (float)
  enable_step_dir = False (bool)
motor:
  error = 0x0000 (int)
  is_calibrated = True (bool)
  timing_log: ...
  current_meas_phB = -0.06507164239883423 (float)
  current_meas_phC = -0.032860398292541504 (float)
  armed_state = 0 (int)
  DC_calib_phB = -0.8210874795913696 (float)
  current_control: ...
  DC_calib_phC = -1.4373878240585327 (float)
  gate_driver: ...
  config: ...
  phase_current_rev_gain = 0.012500000186264515 (float)
trap_traj:
  config: ...
get_temp()
controller:
  error = 0x0000 (int)
  vel_ramp_enable = False (bool)
  vel_ramp_target = 0.0 (float)
  config: ...
  vel_integrator_current = 0.0 (float)
  vel_setpoint = 0.0 (float)
  current_setpoint = 0.0 (float)
  move_to_pos(goal_point: float)
  start_anticogging_calibration()
  pos_setpoint = 0.0 (float)
  set_current_setpoint(current_setpoint: float)
  set_vel_setpoint(vel_setpoint: float, current_feed_forward: float)
  set_pos_setpoint(pos_setpoint: float, vel_feed_forward: float, current_feed_forward: float)
sensorless_estimator:
  vel_estimate = -5.5676350593566895 (float)
  error = 0x0000 (int)
  phase = -2.1344687938690186 (float)
  config: ...
  pll_pos = -2.1973843574523926 (float)
step_dir_active = False (bool)
requested_state = 0 (int)
encoder:
  error = 0x0000 (int)
  is_ready = True (bool)
  index_found = True (bool)
  pos_estimate = 67.98983764648438 (float)
  hall_state = 0 (int)
  vel_estimate = 0.0 (float)
  count_in_cpr = 67 (int)
  interpolation = 0.5 (float)
  pos_cpr = 67.97500610351562 (float)
  config: ...
  shadow_count = 67 (int)
  phase = 0.4278993606567383 (float)
  
  
  
  
odrv0.axis0.motor 
  
error = 0x0000 (int)
is_calibrated = True (bool)
timing_log:
  TIMING_LOG_IDX_SEARCH = 7502 (int)
  TIMING_LOG_MEAS_L = 7298 (int)
  TIMING_LOG_ENC_CALIB = 7450 (int)
  TIMING_LOG_FOC_CURRENT = 0 (int)
  TIMING_LOG_MEAS_R = 7330 (int)
  TIMING_LOG_ADC_CB_I = 2830 (int)
  TIMING_LOG_ADC_CB_DC = 12886 (int)
  TIMING_LOG_GENERAL = 0 (int)
  TIMING_LOG_FOC_VOLTAGE = 7390 (int)
current_meas_phB = 0.03486299514770508 (float)
current_meas_phC = -0.04488956928253174 (float)
armed_state = 0 (int)
DC_calib_phB = -0.8201173543930054 (float)
current_control:
  final_v_beta = 0.0 (float)
  Iq_setpoint = 0.0 (float)
  v_current_control_integral_q = 0.0 (float)
  i_gain = 33.37803268432617 (float)
  v_current_control_integral_d = 0.0 (float)
  Ibus = 0.0 (float)
  Iq_measured = 0.0 (float)
  max_allowed_current = 30.375 (float)
  p_gain = 0.05533279478549957 (float)
  overcurrent_trip_level = 33.75 (float)
  final_v_alpha = 0.0 (float)
DC_calib_phC = -1.4255393743515015 (float)
gate_driver:
  drv_fault = 0 (int)
config:
  current_control_bandwidth = 100.0 (float)
  phase_inductance = 0.0005533279618248343 (float)
  motor_type = 0 (int)
  pre_calibrated = True (bool)
  direction = 1 (int)
  resistance_calib_max_voltage = 4.0 (float)
  calibration_current = 8.0 (float)
  requested_current_range = 25.0 (float)
  current_lim = 10.0 (float)
  pole_pairs = 5 (int)
  phase_resistance = 0.33378034830093384 (float)
phase_current_rev_gain = 0.012500000186264515 (float

In [218]: odrv0.axis0.controller                                                
Out[218]: 
error = 0x0000 (int)
vel_ramp_enable = False (bool)
vel_ramp_target = 0.0 (float)
config:
  control_mode = 2 (int)
  vel_limit = 40000.0 (float)
  vel_ramp_rate = 10000.0 (float)
  vel_integrator_gain = 0.10000000149011612 (float)
  setpoints_in_cpr = False (bool)
  vel_limit_tolerance = 1.2000000476837158 (float)
  pos_gain = 1.0 (float)
  vel_gain = 0.019999999552965164 (float)
vel_integrator_current = 0.0 (float)
vel_setpoint = 0.0 (float)
current_setpoint = 0.0 (float)
move_to_pos(goal_point: float)
start_anticogging_calibration()
pos_setpoint = 257.9796447753906 (float)
set_current_setpoint(current_setpoint: float)
set_vel_setpoint(vel_setpoint: float, current_feed_forward: float)
set_pos_setpoint(pos_setpoint: float, vel_feed_forward: float, current_feed_forward: float)

What makes you say that? Your gains are huge!

Thanks for your answer.

I took those values from the hoverboard guide.

Now I started from scratch and after some experiments these below seem to work although I have one issue.
The current settings are:

  	my_drive.axis0.controller.config.pos_gain = 1
	my_drive.axis0.controller.config.setpoints_in_cpr =  False
	my_drive.axis0.controller.config.vel_gain = 0.0005000000237487257
	my_drive.axis0.controller.config.vel_integrator_gain = 0.001
	my_drive.axis0.controller.config.vel_limit = 50000
	my_drive.axis0.controller.config.vel_limit_tolerance = 100
	my_drive.axis0.controller.config.vel_ramp_rate = 50.0

Are those values good enough, or do you recommend some tuning yet?

When I set both motors to some higher vel_setpoint, then I get motor error 64. What parameter could cause this issue?

And one more question : what should I change to have smooth and stable running at low velocities?

I think you should follow the hoverboard guide for the .motor parameters, but start with the defaults for the .encoder and .controller parameters. And tune from there.