fix(core) Give using unpack for bytes conversion

Now bin2hex and hexdec are used instead, so hexadecimal is used as an
 intermediate for the conversion.
Also, fix a bug that max was never returned.
Also fixed filter which was turning result negative in some cases.

issue #5843