Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 5.885914754s FAIL (2025-07-09 21:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.885939842s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 5.965448067s FAIL (2025-07-09 21:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.965479397s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X.X...XXX.........X......X....X..X.........X...............] duration: 34.587099489s FAIL (2025-07-09 21:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.587120228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X..X..........X...X................X....X...............] duration: 33.754516802s FAIL (2025-07-09 21:31:34 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 33.754560664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 6.05469103s FAIL (2025-07-09 21:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.054710436s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X.X...XXX.........X......X....X..X.........X..............] duration: 57.634197173s FAIL (2025-07-09 21:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.634220297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X....X..X..........X...X................X....X..............] duration: 37.71350779s FAIL (2025-07-09 21:30:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 37.713534321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 5.545246937s FAIL (2025-07-09 21:30:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.545263309s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..X............................................] duration: 6.602217554s FAIL (2025-07-09 21:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.602239747s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...X.X...XXX.........X......X....X..X.........X.............] duration: 26.898548263s FAIL (2025-07-09 21:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.898598889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X....X..X..........X...X................X....X.............] duration: 35.748653457s FAIL (2025-07-09 21:29:38 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 35.748675168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X....X..X..........X...X................X....X............] duration: 54.53547648s FAIL (2025-07-09 21:29:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 54.53550198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....X.X...XXX.........X......X....X..X.........X............] duration: 29.775208182s FAIL (2025-07-09 21:28:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.775223383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration: 5.91617624s FAIL (2025-07-09 21:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.916198604s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.....X.X...XXX.........X......X....X..X.........X...........] duration: 42.535761705s FAIL (2025-07-09 21:28:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.535784891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration: 6.129738445s FAIL (2025-07-09 21:27:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129783421s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..X....X..X..........X...X................X....X...........] duration: 21.743439973s FAIL (2025-07-09 21:27:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 21.743461594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X.X...XXX.........X......X....X..X.........X..........] duration: 26.675500628s FAIL (2025-07-09 21:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.675521889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 6.002983032s FAIL (2025-07-09 21:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.002999984s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......X.X...XXX.........X......X....X..X.........X.........] duration: 38.921155238s FAIL (2025-07-09 21:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.921167322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X..X....X..X..........X...X................X....X..........] duration: 32.289071023s FAIL (2025-07-09 21:26:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 32.289095138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..X........................................] duration: 6.254598424s FAIL (2025-07-09 21:26:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.25462309s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X.X...XXX.........X......X....X..X.........X........] duration: 35.407520447s FAIL (2025-07-09 21:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.407544713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X..X....X..X..........X...X................X....X.........] duration: 18.772978197s FAIL (2025-07-09 21:25:51 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 18.772997273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X..X..X....X..X..........X...X................X....X........] duration: 47.013752722s FAIL (2025-07-09 21:25:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 47.013781438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration: 10.987160835s FAIL (2025-07-09 21:25:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.9871841s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X........X.X...XXX.........X......X....X..X.........X.......] duration: 18.217312507s FAIL (2025-07-09 21:25:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.217324861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..X......................................] duration: 10.075431685s FAIL (2025-07-09 21:24:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.075457305s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X........X.X...XXX.........X......X....X..X.........X......] duration: 26.942692694s FAIL (2025-07-09 21:24:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.942709918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..X..X....X..X..........X...X................X....X.......] duration: 41.681680508s FAIL (2025-07-09 21:24:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 41.681699857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X.X...XXX.........X......X....X..X.........X.....] duration: 46.831464119s FAIL (2025-07-09 21:24:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.831487704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration: 12.916587124s FAIL (2025-07-09 21:24:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.916607663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..X..X....X..X..........X...X................X....X......] duration: 40.165728953s FAIL (2025-07-09 21:23:51 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 40.165745804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X.X...XXX.........X......X....X..X.........X....] duration: 47.223190545s FAIL (2025-07-09 21:23:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.223213307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..X....................................] duration: 6.468368551s FAIL (2025-07-09 21:23:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.468408857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X..X..X....X..X..........X...X................X....X.....] duration: 53.035115311s FAIL (2025-07-09 21:23:11 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 53.035134628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X.X...XXX.........X......X....X..X.........X...] duration: 32.374234996s FAIL (2025-07-09 21:22:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.374246919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X..X..X..X....X..X..........X...X................X....X....] duration: 49.449723236s FAIL (2025-07-09 21:22:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 49.44974612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration: 6.656609644s FAIL (2025-07-09 21:22:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.656633279s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X........X.X...XXX.........X......X....X..X.........X..] duration: 50.609684455s FAIL (2025-07-09 21:22:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.609697379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..X..................................] duration: 6.287863276s FAIL (2025-07-09 21:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.287892051s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......X........X.X...XXX.........X......X....X..X.........X.] duration: 45.815545079s FAIL (2025-07-09 21:21:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.815571228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X..X..X..X....X..X..........X...X................X....X...] duration: 32.427786823s FAIL (2025-07-09 21:20:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 32.427822069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration: 6.299923155s FAIL (2025-07-09 21:20:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299940108s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......X........X.X...XXX.........X......X....X..X.........X] duration: 1m0.001114954s FAIL (2025-07-09 21:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001128348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X..X..X..X..X....X..X..........X...X................X....X..] duration: 55.09454734s FAIL (2025-07-09 21:20:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 55.094580873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..X................................] duration: 6.094836013s FAIL (2025-07-09 21:19:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.094891989s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration: 5.952045828s FAIL (2025-07-09 21:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.952065675s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X........X.X...XXX.........X......X....X..X.........] duration: 38.462532291s FAIL (2025-07-09 21:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.462552271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration: 6.522862551s FAIL (2025-07-09 21:19:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.522900666s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)