Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.....X.X...XXX.........X......X....X..X.........X...........] duration: 42.535761705s FAIL (2025-07-09 21:28:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.535784891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration: 6.129738445s FAIL (2025-07-09 21:27:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129783421s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..X....X..X..........X...X................X....X...........] duration: 21.743439973s FAIL (2025-07-09 21:27:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 21.743461594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X.X...XXX.........X......X....X..X.........X..........] duration: 26.675500628s FAIL (2025-07-09 21:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.675521889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 6.002983032s FAIL (2025-07-09 21:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.002999984s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......X.X...XXX.........X......X....X..X.........X.........] duration: 38.921155238s FAIL (2025-07-09 21:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.921167322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X..X....X..X..........X...X................X....X..........] duration: 32.289071023s FAIL (2025-07-09 21:26:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 32.289095138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..X........................................] duration: 6.254598424s FAIL (2025-07-09 21:26:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.25462309s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X.X...XXX.........X......X....X..X.........X........] duration: 35.407520447s FAIL (2025-07-09 21:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.407544713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X..X....X..X..........X...X................X....X.........] duration: 18.772978197s FAIL (2025-07-09 21:25:51 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 18.772997273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X..X..X....X..X..........X...X................X....X........] duration: 47.013752722s FAIL (2025-07-09 21:25:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 47.013781438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration: 10.987160835s FAIL (2025-07-09 21:25:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.9871841s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X........X.X...XXX.........X......X....X..X.........X.......] duration: 18.217312507s FAIL (2025-07-09 21:25:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.217324861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..X......................................] duration: 10.075431685s FAIL (2025-07-09 21:24:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.075457305s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X........X.X...XXX.........X......X....X..X.........X......] duration: 26.942692694s FAIL (2025-07-09 21:24:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.942709918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..X..X....X..X..........X...X................X....X.......] duration: 41.681680508s FAIL (2025-07-09 21:24:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 41.681699857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X.X...XXX.........X......X....X..X.........X.....] duration: 46.831464119s FAIL (2025-07-09 21:24:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.831487704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration: 12.916587124s FAIL (2025-07-09 21:24:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.916607663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..X..X....X..X..........X...X................X....X......] duration: 40.165728953s FAIL (2025-07-09 21:23:51 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 40.165745804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X.X...XXX.........X......X....X..X.........X....] duration: 47.223190545s FAIL (2025-07-09 21:23:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.223213307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..X....................................] duration: 6.468368551s FAIL (2025-07-09 21:23:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.468408857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X..X..X....X..X..........X...X................X....X.....] duration: 53.035115311s FAIL (2025-07-09 21:23:11 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 53.035134628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X.X...XXX.........X......X....X..X.........X...] duration: 32.374234996s FAIL (2025-07-09 21:22:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.374246919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X..X..X..X....X..X..........X...X................X....X....] duration: 49.449723236s FAIL (2025-07-09 21:22:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 49.44974612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration: 6.656609644s FAIL (2025-07-09 21:22:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.656633279s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X........X.X...XXX.........X......X....X..X.........X..] duration: 50.609684455s FAIL (2025-07-09 21:22:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.609697379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..X..................................] duration: 6.287863276s FAIL (2025-07-09 21:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.287892051s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......X........X.X...XXX.........X......X....X..X.........X.] duration: 45.815545079s FAIL (2025-07-09 21:21:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.815571228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X..X..X..X....X..X..........X...X................X....X...] duration: 32.427786823s FAIL (2025-07-09 21:20:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 32.427822069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration: 6.299923155s FAIL (2025-07-09 21:20:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299940108s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......X........X.X...XXX.........X......X....X..X.........X] duration: 1m0.001114954s FAIL (2025-07-09 21:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001128348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X..X..X..X..X....X..X..........X...X................X....X..] duration: 55.09454734s FAIL (2025-07-09 21:20:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 55.094580873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..X................................] duration: 6.094836013s FAIL (2025-07-09 21:19:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.094891989s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration: 5.952045828s FAIL (2025-07-09 21:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.952065675s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X........X.X...XXX.........X......X....X..X.........] duration: 38.462532291s FAIL (2025-07-09 21:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.462552271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration: 6.522862551s FAIL (2025-07-09 21:19:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.522900666s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X..X..X..X..X....X..X..........X...X................X....X.] duration: 41.890530196s FAIL (2025-07-09 21:18:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 41.890550043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..X.............................] duration: 5.882373979s FAIL (2025-07-09 21:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.882397755s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X..X..X..X..X....X..X..........X...X................X....X] duration: 1m0.003000998s FAIL (2025-07-09 21:18:04 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 1m0.003070721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..X............................] duration: 13.034947671s FAIL (2025-07-09 21:18:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.034970635s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X........X........X.X...XXX.........X......X....X..X........] duration: 35.223271227s FAIL (2025-07-09 21:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.223284914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..X...........................] duration: 6.110389279s FAIL (2025-07-09 21:17:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.110461127s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X........X........X.X...XXX.........X......X....X..X.......] duration: 45.609517399s FAIL (2025-07-09 21:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.609529081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X..X..X..X..X..X....X..X..........X...X................X....] duration: 34.945520492s FAIL (2025-07-09 21:17:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 34.945536713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X........X.X...XXX.........X......X....X..X......] duration: 36.478329797s FAIL (2025-07-09 21:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.478343553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X..X..X..X..X..X....X..X..........X...X................X...] duration: 45.095442657s FAIL (2025-07-09 21:16:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 45.095461744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration: 5.937318399s FAIL (2025-07-09 21:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.937342754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X........X.X...XXX.........X......X....X..X.....] duration: 24.756651295s FAIL (2025-07-09 21:16:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.756673989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration: 5.942329991s FAIL (2025-07-09 21:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.942352472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X..X..X..X..X..X....X..X..........X...X................X..] duration: 37.025428265s FAIL (2025-07-09 21:15:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 37.025447662s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)