Keep the helical focuser.
Remove the nosepiece and I/R filter from the ASI178. Thread the filter into the internal threads directly in front of the sensor.
Remove the 1 1/4 nosepiece thumb screws on the helical focuser.
Thread the camera all the way on to the focuser and check orientation to the OAG prism. It should line up! If it does not, get a very thin m42 threaded stop ring and thread it all the way on to the helical focuser then thread on the camera and back the camera off only enough to get it oriented with the prism. Tighten the stop ring against the camera to secure it to the focuser. Set the helical focuser to mid point (+4 mm).
Note, the back focus of the ASI178 is split. The sensor is set 5 mm behind the front surface of the sensor window. The depth of the M42 threaded nose of the camera is 7.50 mm.
If the helical focuser's M42 threads are 7.5mm or more and the camera threads completely on, the minimum backfocus of the helical focuser plus camera is 37 +5 + = 42 mm. Maximum focus distance is 45 +5 = 50mm. If a stop ring is used or the focuser threads are not deep enough you will need to measure this distance: The minimum depth of the focuser from the OAG frame to the front most edge of the camera and add 12.5 mm to get the this distance and add 8mm more to get maximum (8mm helical focuser range)
The last adjustment is the prism depth. It looks like the prism tube is fully extended. (6+14+6=26 mm). Focus distance (6+14 +0) = 20 mm.
Focus range *fully threaded camera on helical focuser* is
62 to 70 mm ( you can march focus with the main camera at 65 mm)
However, the prism looks a bit low and may interfere with main sensor. Adjust it
Loosen 2mm set screw(s) on the OAG frame that lock the prism in place. This may be in on the camera side or may be on the OAG frame edge. BE CAREFUL. the prism carrier will be loose and can easily fall out. Do NOT TOUCH the prism. Surface!
Move the prism further away from center by 3 mm (shaft measurement now 11mm). This puts the center of the focus range right at 65 mm.
Focus main camera 1st. Then guide camera.
Check for vignetting on both the main camera and guide camera. (During day)
Main camera vignetting, move prism away from center and refocus guide cam.
Guide camera vignetting, move prism towards center. Refocus Guide cam.
Tips:
Be sure prism is perpendicular to the focal plane when adjusting length. Prism may extend into guide camera. if adjusted to high (large main sensors) the top of prism may hit filter or sensor. BE CAREFUL.
. If you have tilt adjustments, take test shots and adjust out any tilt. Tilt set screws can also be used as fine adjustment of Guide camera backfocus avoiding the 42mm stop ring. Similarly some OAGs have tilt/stop screws to adjust the stop position of the helical focuser where it threads on to OAG .
A note about your prism backfocus measurement point.
The image seen by the guider is refracted 90° by the 45° prism surface. The measurement point for focus is the plane of the image, which is the closest point to the camera where the 45° surface starts, not the midpoint of the slope.
Be sure everything is tight and lock the helical focuser. You should not need to adjust OAG focus again!
Worth the effort..
If you switch to a DSLR the flange to sensor distance can vary slightly. Cannon EF is 44mm sensor to flange plus 11mm Flange to T2 adapter = 55 mm.
Mirrorless is only 17.5 mm. to flange. However, if your buy the Cannon mirrorless camera T adapter it adds the 17.5 mm back in (it's 37.5 mm) to maintain the industry 55 mm standard.