Text Detection and recognition in Android using OpenCV MSER algorithm

For Real Time text detection in Android, we will be using OpenCV, and to recognize the text in from the detected text region, we will use Tesseract text detection engine for Android. This part deals with the text detection using OpenCV for Android part. In the next part we will recognize the detected text using openCV.

Android Text Detection using MSER Algorithm

To detect the text region, we will use MSER algorithm using OpenCV for Android. MSER stands for maximally stable extremal regions and is a very popular method used for blob detection from images. We will be using MSER as it detects many key characteristics of the region under study and can be used to study very small regions too.
 

Steps to detect text region in android using OpenCV (MSER algorithm) are :  

1. Create a layout containing JavaCameraView 
2. Create an Activity that implements CameraBridgeViewBase.CvCameraViewListener 
3. Perform the Text Detection inside the OnCameraFrame 

Text being detected from an image of bill using Android Camera
Text Detection from an Image of a Bill

First let us create a Layout that will contain the JavaCameraView, Here is the code for main_layout containing javaCameraView –

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

 <org.opencv.android.JavaCameraView
        android:id="@+id/my_java_camera"
        android:layout_width="match_parent1"
        android:layout_height="match_parent" />

</LinearLayout>

Now our layout is done, but there will be orientation problem, learn here – how to fix it and show full screen camera. Now we need an activity, here is the MainActivity for text detection in Android using openCV

public class MainActivity extends AppCompatActivity implements CameraBridgeViewBase.CvCameraViewListener2{
  private Mat mGrey, mRgba;
  private JavaCameraView mOpenCvCameraView;
private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
                case LoaderCallbackInterface.SUCCESS: {
                    mOpenCvCameraView.enableView();
                }
                break;
                default: {
                    super.onManagerConnected(status);
                }
                break;
            }
        }
    };
@Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        //FULLSCREEN MODE
        getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,      WindowManager.LayoutParams.FLAG_FULLSCREEN);
        getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
        setContentView(R.layout.main_layout);

        mOpenCvCameraView = (MTCameraView) findViewById(R.id.my_java_camera);
        mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);
        mOpenCvCameraView.setCvCameraViewListener(this);

    }

public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
        mGrey = inputFrame.gray();
        mRgba = inputFrame.rgba();

        detectText();
        return mRgba;
    }
@Override
    public void onPause() {
        super.onPause();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }

    @Override
    public void onResume() {
        super.onResume();
        if (!OpenCVLoader.initDebug()) {
            OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_3_1_0, this, mLoaderCallback);
        } else {
            mLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);
        }
    }

    public void onDestroy() {
        super.onDestroy();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }

    public void onCameraViewStarted(int width, int height) {
        mIntermediateMat = new Mat();
        mGrey = new Mat(height, width, CvType.CV_8UC4);
        mRgba = new Mat(height, width, CvType.CV_8UC4);
    }

private void detectText() {
        MatOfKeyPoint keypoint = new MatOfKeyPoint();
        List<KeyPoint> listpoint;
        KeyPoint kpoint;
        Mat mask = Mat.zeros(mGrey.size(), CvType.CV_8UC1);
        int rectanx1;
        int rectany1;
        int rectanx2;
        int rectany2;
        int imgsize = mGrey.height() * mGrey.width();
        Scalar zeos = new Scalar(0, 0, 0);

        List<MatOfPoint> contour2 = new ArrayList<MatOfPoint>();
        Mat kernel = new Mat(1, 50, CvType.CV_8UC1, Scalar.all(255));
        Mat morbyte = new Mat();
        Mat hierarchy = new Mat();

        Rect rectan3;
        //
        FeatureDetector detector = FeatureDetector
                .create(FeatureDetector.MSER);
        detector.detect(mGrey, keypoint);
        listpoint = keypoint.toList();
        //
        for (int ind = 0; ind < listpoint.size(); ind++) {
            kpoint = listpoint.get(ind);
            rectanx1 = (int) (kpoint.pt.x - 0.5 * kpoint.size);
            rectany1 = (int) (kpoint.pt.y - 0.5 * kpoint.size);
            rectanx2 = (int) (kpoint.size);
            rectany2 = (int) (kpoint.size);
            if (rectanx1 <= 0)
                rectanx1 = 1;
            if (rectany1 <= 0)
                rectany1 = 1;
            if ((rectanx1 + rectanx2) > mGrey.width())
                rectanx2 = mGrey.width() - rectanx1;
            if ((rectany1 + rectany2) > mGrey.height())
                rectany2 = mGrey.height() - rectany1;
            Rect rectant = new Rect(rectanx1, rectany1, rectanx2, rectany2);
            try {
                Mat roi = new Mat(mask, rectant);
                roi.setTo(CONTOUR_COLOR);
            } catch (Exception ex) {
                Log.d("mylog", "mat roi error " + ex.getMessage());
            }
        }

        Imgproc.morphologyEx(mask, morbyte, Imgproc.MORPH_DILATE, kernel);
        Imgproc.findContours(morbyte, contour2, hierarchy,
                Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_NONE);
        for (int ind = 0; ind < contour2.size(); ind++) {
            rectan3 = Imgproc.boundingRect(contour2.get(ind));
            rectan3 = Imgproc.boundingRect(contour2.get(ind));
            if (rectan3.area() > 0.5 * imgsize || rectan3.area() < 100
                    || rectan3.width / rectan3.height < 2) {
                Mat roi = new Mat(morbyte, rectan3);
                roi.setTo(zeos);

            } else
                Imgproc.rectangle(mRgba, rectan3.br(), rectan3.tl(),
                        CONTOUR_COLOR);
        }
    }

}

And we our done! Bravo. Now just declare the activity in the manifest and run it, you will be able to see our app detecting various regions. It’s our job to decide which one is the appropriate text region. In the code above to, the text detection method is detectText(); it contains initialization of feature detector for MSER. So now you can see that our app is capable of detecting region. In the next part, we will seee how we can extract those regions and pass it to Tesseract for text Recognition. Stay tuned to this guide on text detection in Android Using OpenCV, the second part will is out! Check out – Text detection using Tesseract.
Happy coding, do ask any questions that you have below or reach me out on social media platforms.

Don’t miss these tips!

We don’t spam! Read our privacy policy for more info.

Sharing is caring!

Leave a Comment

Your email address will not be published.