You were just not driven enough...I played football thru HS and College even on the Army regional team...was just 5'7" 150lbs at best weight...not big enough in college to play offense ,but I could take the offense out playing deepback/corner defense! However I must admitt I was way too small to even think about pro...those dudes are big,bad and onery to say the least!
However for your comment about speed...I did run the 100 yd in 10.5...so I could catch most offense and did a mean run back on kick-offs/punts!