Beagleboard physical design

Has anyone re-designed Beagleboard in different form factor to package
in custom enclosure? If so, were there any challenges in converting
Orcad schematic to the board layout?

If custom board layout is required, any suggestions on do's and
dont's. Thanks.

Do it right. Don’t do it it wrong. I would keep the core layout of the Beagle and add your stuff around it. A lot of people have redone the layout, but most of them have kept the core layout.

OrCad generates netlist for all sorts of layout packages. But, you will be starting over on the layout. Allegro is the layout package that it is in. There is an Altium version available to start from as well as Cadstar.