Setting up osTicket on cPanel is easy and somewhat in Plesk as well, especially the email piping part as they have in depth articles on their wiki. But as soon as you try to set up email piping other than cPanel hosting and Plesk hosting environments, you are in the uncharted waters. Recently, I had to set up osTicket support system on Joyent Smart Machine server and this is how I got it done.
Inside Joyent Smart Machine hosting Admin Panel, you should be able to create mail aliases using Postfix Mail Server that will direct the e-mails to a /home/user-account/web/public/osTicket/api/pipe.php script.
Go to Virtualmin > Your Server > Postfix Mail Server > Edit Mail Aliases > Add an alias
and provide information for these following fields:
Description, Address(email), Enabled? Yes/No and Alias to select “Feed to program ..” from the drop-down, and in the space provided, provide full path to the osTicket mail piping script /home/user-account/web/public/osTicket-dir/api/pipe.php
Make sure that you set the executable permissions (chmod 755) on pipe.php otherwise it might not run.
Also, If you were to make the directories leading up to your script executable, you could also rule out permission issue by something like:
chmod +x /home/user-account/web/public/osTicket-dir/api/
Hope that helps.