A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 8 Issue 8
Aug.  2021

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 6.171, Top 11% (SCI Q1)
    CiteScore: 11.2, Top 5% (Q1)
    Google Scholar h5-index: 51, TOP 8
Turn off MathJax
Article Contents
Kunhua Liu, Zihao Ye, Hongyan Guo, Dongpu Cao, Long Chen and Fei-Yue Wang, "FISS GAN: A Generative Adversarial Network for Foggy Image Semantic Segmentation," IEEE/CAA J. Autom. Sinica, vol. 8, no. 8, pp. 1428-1439, Aug. 2021. doi: 10.1109/JAS.2021.1004057
Citation: Kunhua Liu, Zihao Ye, Hongyan Guo, Dongpu Cao, Long Chen and Fei-Yue Wang, "FISS GAN: A Generative Adversarial Network for Foggy Image Semantic Segmentation," IEEE/CAA J. Autom. Sinica, vol. 8, no. 8, pp. 1428-1439, Aug. 2021. doi: 10.1109/JAS.2021.1004057

FISS GAN: A Generative Adversarial Network for Foggy Image Semantic Segmentation

doi: 10.1109/JAS.2021.1004057
Funds:  This work was supported in part by the National Key Research and Development Program of China (2018YFB1305002), the National Natural Science Foundation of China (62006256), the Postdoctoral Science Foundation of China (2020M683050), the Key Research and Development Program of Guangzhou (202007050002), and the Fundamental Research Funds for the Central Universities (67000-31610134)
More Information
  • Because pixel values of foggy images are irregularly higher than those of images captured in normal weather (clear images), it is difficult to extract and express their texture. No method has previously been developed to directly explore the relationship between foggy images and semantic segmentation images. We investigated this relationship and propose a generative adversarial network (GAN) for foggy image semantic segmentation (FISS GAN), which contains two parts: an edge GAN and a semantic segmentation GAN. The edge GAN is designed to generate edge information from foggy images to provide auxiliary information to the semantic segmentation GAN. The semantic segmentation GAN is designed to extract and express the texture of foggy images and generate semantic segmentation images. Experiments on foggy cityscapes datasets and foggy driving datasets indicated that FISS GAN achieved state-of-the-art performance.

     

  • loading
  • [1]
    L. Chen, W. J. Zhan, W. Tian, Y. H. He, and Q. Zou, “Deep integration: A multi-label architecture for road scene recognition,” IEEE Trans. Image Process., vol. 28, no. 10, pp. 4883–4898, Oct. 2019. doi: 10.1109/TIP.2019.2913079
    [2]
    K. Wada, K. Okada, and M. Inaba, “Joint learning of instance and semantic segmentation for robotic pick-and-place with heavy occlusions in clutter,” in Proc. IEEE Int. Conf. Robotics and Autom., Montreal, Canada, 2019, pp. 9558–9564.
    [3]
    Y. C. Ouyang, L. Dong, L. Xue and C. Y. Sun, “Adaptive control based on neural networks for an uncertain 2-DOF helicopter system with input deadzone and output constraints,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 3, pp. 807–815, May 2019. doi: 10.1109/JAS.2019.1911495
    [4]
    Y. H. Luo, S. N. Zhao, D. S. Yang, and H. W. Zhang, “A new robust adaptive neural network backstepping control for single machine infinite power system with TCSC,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 1, pp. 48–56, Jan. 2020.
    [5]
    N. Zerari, M. Chemachema, and N. Essounbouli, “Neural network based adaptive tracking control for a class of pure feedback nonlinear systems with input saturation,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 278–290, Jan. 2019. doi: 10.1109/JAS.2018.7511255
    [6]
    D. Wu and X. Luo, “Robust latent factor analysis for precise representation of high-dimensional and sparse data,” IEEE/CAA J. Autom. Sinica, pp. 766–805, Dec. 2019.
    [7]
    X. Luo, Y. Yuan, S. L. Chen, N. Y. Zeng, and Z. D. Wang, “Position-transitional particle swarm optimization-incorporated latent factor analysis,” IEEE Trans. Knowl. Data. En., pp. 1–1, Oct. 2019.
    [8]
    A. Cantor, “Optics of the atmosphere: Scattering by molecules and particles,” IEEE J. Quantum. Elect., vol. 14, no. 9, pp. 698–699, Sept. 1978.
    [9]
    S. G. Narasimhan and S. K. Nayar, “Vision and the atmosphere,” Int. J. Comput. Vision, vol. 48, no. 3, pp. 233–254, Jul. 2002. doi: 10.1023/A:1016328200723
    [10]
    P. Luc, C. Couprie, S, Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” arXiv preprint arXiv: 1611.08408, Dec. 2016.
    [11]
    A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr, “Higher order conditional random fields in deep neural networks,” in Proc. European Conf. Computer Vision, B. Leibe, J. Matas, N. Sebe and M. Welling Eds. Cham, Germany: Springer, 2016, pp. 524–540.
    [12]
    P. Isola, J. Y. Zhu, T. H. Zhou, and A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE. Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 1125–1134.
    [13]
    J. Hoffman, E. Tzeng, T. Park, Y. J. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in Proc. 35th Int. Conf. Machine Learning, Stockholm, Sweden, 2018, pp. 1989–1998.
    [14]
    K. Nazeri, E. Ng, T. Joseph, F. Z. Qureshi, and M. Ebrahimi, “Edgeconnect: Generative image inpainting with adversarial edge learning,” arXiv preprint arXiv: 1901.00212, Jan. 2019.
    [15]
    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. Int. Conf. Medical Image Computing and Computer-assisted Intervention. Cham, Germany: Springer, 2015, pp. 234–241.
    [16]
    J. Y. Kim, L. S. Kim, and S. H. Hwang, “An advanced contrast enhancement using partially overlapped sub-block histogram equalization,” in Proc. Int. Conf. IEEE Symposium on Circuits and Systems, Geneva, Switzerland, 2000, pp. 537–540.
    [17]
    A. Eriksson, G. Capi, and K. Doya, “Evolution of meta-parameters in reinforcement learning algorithm,” in Proc. IEEE/RSJ. Int. Conf. Intelligent Robots and System, Las Vegas, USA, 2003, pp. 412–417.
    [18]
    M. J. Seow and V. K. Asari, “Ratio rule and homomorphic filter for enhancement of digital colour image,” Neurocomputing, vol. 69, no. 7–9, pp. 954–958, Mar. 2006. doi: 10.1016/j.neucom.2005.07.003
    [19]
    S. Shwartz, E. Namer, and Y. Y. Schechner, “Blind haze separation,” in Proc. IEEE. Int. Conf. Computer Vision and Pattern Recognition, New York, USA, 2006, pp. 1984–1991.
    [20]
    Y. Y. Schechner and Y. Averbuch, “Regularized image recovery in scattering media,” IEEE Trans. Pattern Anal., vol. 29, no. 9, pp. 1655–1660, Sept. 2007. doi: 10.1109/TPAMI.2007.1141
    [21]
    R. T. Tan, “Visibility in bad weather from a single image,” in Proc. IEEE. Int. Conf. Computer Vision and Pattern Recognition, Anchorage, USA, 2008, pp. 1–8.
    [22]
    R. Fattal, “Single image dehazing,” ACM Trans. Graphic., vol. 27, no. 3, pp. 1–9, Aug. 2008.
    [23]
    K. M. He, J. Sun, and X. O. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal., vol. 33, no. 12, pp. 2341–2353, Dec. 2011. doi: 10.1109/TPAMI.2010.168
    [24]
    K. B. Gibson and T. Q. Nguyen, “On the effectiveness of the dark channel prior for single image dehazing by approximating with minimum volume ellipsoids,” in Proc. IEEE. Int. Conf. Acoustics, Speech and Signal Processing, Prague, Czech Republic, 2011, pp. 1253–1256.
    [25]
    D. F. Shi, B. Li, W. Ding, and Q. M. Chen, “Haze removal and enhancement using transmittance-dark channel prior based on object spectral characteristic,” Acta Autom. Sinica, vol. 39, no. 12, pp. 2064–2070, Dec. 2013.
    [26]
    S. G. Narasimhan and S. K. Nayar, “Interactive (de) weathering of an image using physical models,” in Proc. IEEE Workshop on color and photometric Methods in computer Vision, vol. 6, no. 4, Article No. 1, Jan. 2003.
    [27]
    S. G. Narasimhan and S. K. Nayar, “Chromatic framework for vision in bad weather,” in Proc. IEEE. Int. Conf. Computer Vision and Pattern Recognition, Hilton Head, USA, 2000, pp. 598–605.
    [28]
    J. Tarel and N. Hautière, “Fast visibility restoration from a single color or gray level image,” in Proc. 12th IEEE Int. Conf. Computer Vision, Kyoto, Japan, 2009, pp. 2201–2208.
    [29]
    H. Zhang, V. Sindagi, and V. M. Patel, “Joint transmission map estimation and dehazing using deep networks,” IEEE Trans. Circ. Syst. Vid., vol. 30, no. 7, Jul. 2020.
    [30]
    W. Q. Ren, S. Liu, H. Zhang, J. S. Pan, X. C. Cao, and M. H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in Proc. European Conf. Computer Vision, B. Leibe, J. Matas, N. Sebe and M. Welling Eds. Cham, Germany: Springer, 2016, pp. 154–169.
    [31]
    H. Zhang and V. M. Patel, “Densely connected pyramid defogging network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 3194–3203.
    [32]
    B. L. Cai, X. M. Xu, K. Jia, C. M. Qing, and D. C. Tao, “DehazeNet: An end-to-end system for single image haze removal,” IEEE Trans. Image Process., vol. 25, no. 11, pp. 5187–5198, Nov. 2016. doi: 10.1109/TIP.2016.2598681
    [33]
    D. D. Chen, M. M. He, Q. N. Fan, J. Liao, L. H. Zhang, D. D. Hou, L. Yuan, and G. Hua, “Gated context aggregation network for image dehazing and deraining,” in Proc. IEEE Winter Conf. Applications of Computer Vision, Waikoloa, USA, 2019, pp. 1375–1383.
    [34]
    S. Y. Huang, H. X. Li, Y. Yang, B. Wang, and N. N. Rao, “An end-to-end dehazing network with transitional convolution layer,” Multidim. Syst. Sign P., vol. 31, no. 4, pp. 1603–1623, Mar. 2020. doi: 10.1007/s11045-020-00723-2
    [35]
    H. Zhang, V. Sindagi, and V. M. Patel, “Multi-scale single image dehazing using perceptual pyramid deep network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018, pp. 902–911.
    [36]
    X. Qin, Z. L. Wang, Y. C. Bai, X. D. Xie, and H. Z. Jia, “FFA-Net: Feature fusion attention network for single image defogging,” arXiv preprint arXiv: 1911.07559, Nov. 2019.
    [37]
    Q. S. Yi, A. W. Jiang, J. C. Li, J. Y. Wan, and M. W. Wang, “Progressive back-traced dehazing network based on multi-resolution recurrent reconstruction,” IEEE Access, vol. 8, pp. 54514–54521, Mar. 2020. doi: 10.1109/ACCESS.2020.2981491
    [38]
    B. Y. Li, X. L. Peng, Z. Y. Wang, J. Z. Xu, and D. Feng, “An all-in-one network for defogging and beyond,” arXiv preprint arXiv: 1707.06543, Jul. 2017.
    [39]
    H. Y. Zhu, X. Peng, V. Chandrasekhar, L. Y. Li, and J. H. Lim, “DehazeGAN: When image dehazing meets differential programming,” in Proc. 27th Int. Joint Conf. Artificial Intelligence, Stockholm, Sweden, 2018, pp. 1234–1240.
    [40]
    R. D. Li, J. S. Pan, Z. C. Li, and J. H. Tang, “Single image dehazing via conditional generative adversarial network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 8202–8211.
    [41]
    D. Engin, A. Genc, and H. K. Ekenel, “Cycle-dehaze: enhanced cycleGAN for single image dehazing,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018, pp. 825–833.
    [42]
    G. Kim, J. Park, S. Ha, and J. Kwon, “Bidirectional deep residual learning for haze removal,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Long Beach Convention and Entertainment Center, USA, 2018, pp. 46–54.
    [43]
    A. Dudhane and S. Murala, “CDNet: Single image de-hazing using unpaired adversarial training,” in Proc. IEEE Winter Conf. Applications of Computer Vision, Waikoloa, USA, 2019, pp. 1147–1155.
    [44]
    P. Sharma, P. Jain, and A. Sur, “Scale-aware conditional generative adversarial network for image defogging,” in Proc. IEEE Winter Conf. Applications of Computer Vision, Snowmass, USA, 2020, pp. 2355–2365.
    [45]
    W. D. Yan, A. Sharma, and R. T. Tan, “Optical flow in dense foggy scenes using semi-supervised learning,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 13259–13268.
    [46]
    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, USA, 2015, pp. 3431–3440.
    [47]
    L. Chen, W. J. Zhan, J. J. Liu, W. Tian, and D. P. Cao, “Semantic segmentation via structured refined prediction and dual global priors,” in Proc. IEEE Int. Conf. Advanced Robotics and Mechatronics, Toyonaka, Japan, 2019, pp. 53–58.
    [48]
    Y. H. Yuan, X. L. Chen, and J. D. Wang, “Object-contextual representations for semantic segmentation,” arXiv preprint arXiv: 1909.11065, Sept. 2019.
    [49]
    S. Choi, J. T. Kim, and J. Choo, “Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 9373–9383.
    [50]
    M. H. Yin, Z. L. Yao, Y. Cao, X. Li, Z. Zhang, S. Lin, and H. Hu, “Disentangled non-local neural networks,” arXiv preprint arXiv: 2006.06668, Sept. 2020.
    [51]
    H. S. Zhao, J. P. Shi, X. J. Qi, X. G. Wang, and J. Y. Jia, “Pyramid scene parsing network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 2881–2890.
    [52]
    L. C. Chen, Y. K. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 801–818.
    [53]
    H. Y. Chen, L. H. Tsai, S. C. Chang, J. Y. Pan, Y. T. Chen, W. Wei, and D. C. Juan, “Learning with hierarchical complement objective,” arXiv preprint arXiv: 1911.07257, Nov, 2019.
    [54]
    J. Fu, J. Liu, H. J. Tian, Y. Li, Y. J. Bao, Z. W. Fang, and H. Q. Lu, “Dual attention network for scene segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 3146–3154.
    [55]
    C. Zhang, G. S. Lin, F. Y. Liu, R. Yao, and C. H. Chen, “Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 5217–5226.
    [56]
    J. J. He, Z. Y. Deng, L. Zhou, Y. L. Wang, and Y. Qiao, “Adaptive pyramid context network for semantic segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 7519–7528.
    [57]
    J. Hoffman, D. Q. Wang, F. Yu, and T. Darrell, “FCNs in the wild: Pixel-level adversarial and constraint-based adaptation,” arXiv preprint arXiv: 1612.02649, Dec, 2016.
    [58]
    Y. H. Zhang, Z. F. Qiu, T. Yao, D. Liu, and T. Mei, “Fully convolutional adaptation networks for semantic segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 6810–6818.
    [59]
    Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim, “Image to image translation for domain adaptation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 4500–4509.
    [60]
    W. X. Hong, Z. Z. Wang, M. Yang, and J. S. Yuan, “Conditional generative adversarial network for structured domain adaptation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 1335–1344.
    [61]
    Y. W. Luo, L. Zheng, T. Guan, J. Q. Yu, and Y. Yang, “Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 2502–2511.
    [62]
    Y. S. Li, L. Yuan, and N. Vasconcelos, “Bidirectional learning for domain adaptation of semantic segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 6936–6945.
    [63]
    J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal., vol. PAMI-8, no. 6, pp. 679–698, Nov. 1986. doi: 10.1109/TPAMI.1986.4767851
    [64]
    J. Y. Zhu, T. Park, P. Isola and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 2223–2232.
    [65]
    C. Sakaridis, D. X. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” Int. J. Comput. Vision, vol. 126, no. 9, pp. 973–992, Mar. 2018. doi: 10.1007/s11263-018-1072-8
    [66]
    V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proc. 27th Int. Conf. on Int. Conf. on Machine Learning, F. Johannes and J. Thorsten, Eds, 2010, pp. 807–814.
    [67]
    A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in Proc. ICML, vol. 30, no. 1, pp. 3, 2013.
    [68]
    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv: 1412.6980, Dec. 2014.

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(9)  / Tables(6)

    Article Metrics

    Article views (793) PDF downloads(69) Cited by()

    Highlights

    • No method has previously been developed to directly explore the relationship between foggy images and semantic segmentation images. We investigated this relationship and propose a generative adversarial network (GAN) for foggy image semantic segmentation.
    • We propose a novel efficient network architecture based on a combination of concepts from U_Net, called a dilated convolution U_Net. By incorporating dilated convolution layers and adjusting the feature size in the convolutional layer, dilated convolution U_Net has shown improved feature extraction and expression ability.
    • A direct FISS GAN that generates semantic segmentation images under edge information guidance is proposed. We show our method’s effectiveness through extensive experiments on foggy cityscapes datasets and foggy driving datasets and achieve state-of-the-art performance.

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return