Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X........X.......XX..X..XX..X.X.........XX.XX.............] duration: 3.796944042s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:58:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.796983277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.....X.........X.X.....X.......X......X............X.......] duration: 6.040146866s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:58:10 UTC) orbiter warning: context deadline exceeded Duration: 6.040176602s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX.X.........X........] duration: 18.900911581s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 18.900940636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.....X.........X.X.....X.......X......X............X......] duration: 5.898947988s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:57:47 UTC) orbiter warning: context deadline exceeded Duration: 5.898982925s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX.X.........X.......] duration: 15.896711161s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:57:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.896739986s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X........X.......XX..X..XX..X.X.........XX.XX............] duration: 3.484078851s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:57:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484098148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.....X.........X.X.....X.......X......X............X.....] duration: 6.078270298s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:56:55 UTC) orbiter warning: context deadline exceeded Duration: 6.078329852s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.....X.........X.X.....X.......X......X............X....] duration: 12.007242398s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:56:45 UTC) orbiter warning: context deadline exceeded Duration: 12.00732206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX.X.........X......] duration: 16.023835981s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 16.023865747s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....X........X.......XX..X..XX..X.X.........XX.XX...........] duration: 3.556001685s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:56:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556032926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.....X.........X.X.....X.......X......X............X...] duration: 18.832858938s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:56:23 UTC) orbiter warning: context deadline exceeded Duration: 18.832923711s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X........X.......XX..X..XX..X.X.........XX.XX..........] duration: 3.851352334s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:55:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.851380519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX.X.........X.....] duration: 15.152715953s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:55:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.152756442s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.....X.........X.X.....X.......X......X............X..] duration: 19.745037184s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:55:26 UTC) orbiter warning: context deadline exceeded Duration: 19.745064105s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X........X.......XX..X..XX..X.X.........XX.XX.........] duration: 4.273967818s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:54:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.274044814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX.X.........X....] duration: 14.721742266s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:54:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 14.721759028s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.....X.........X.X.....X.......X......X............X.] duration: 30.001816132s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:54:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001838515s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X........X.......XX..X..XX..X.X.........XX.XX........] duration: 3.464598207s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:54:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464627693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX.X.........X...] duration: 15.048408865s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.048437791s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.....X.........X.X.....X.......X......X............X] duration: 30.001336047s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:53:56 UTC) orbiter warning: context deadline exceeded Duration: 30.001457949s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX.X.........X..] duration: 15.521335712s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.521358265s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........X........X.......XX..X..XX..X.X.........XX.XX.......] duration: 4.145861122s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:53:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.145883504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.....X.........X.X.....X.......X......X............] duration: 6.058411926s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:53:01 UTC) orbiter warning: context deadline exceeded Duration: 6.058451621s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.....X.........X.X.....X.......X......X...........] duration: 6.117731501s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:52:41 UTC) orbiter warning: context deadline exceeded Duration: 6.117751468s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X........X.......XX..X..XX..X.X.........XX.XX......] duration: 5.001403956s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:52:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001444492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X.....X.........X.X.....X.......X......X..........] duration: 24.057749499s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:52:35 UTC) orbiter warning: context deadline exceeded Duration: 24.05886345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX.X.........X.] duration: 15.290518385s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:52:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.290547251s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........X........X.......XX..X..XX..X.X.........XX.XX.....] duration: 3.461227379s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:51:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461258217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X.....X.........X.X.....X.......X......X.........] duration: 6.798578302s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:51:45 UTC) orbiter warning: context deadline exceeded Duration: 6.79863478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX.X.........X] duration: 30.640635084s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:51:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.640683667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...........X........X.......XX..X..XX..X.X.........XX.XX....] duration: 3.383034793s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:51:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383069291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X.....X.........X.X.....X.......X......X........] duration: 7.017886066s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:51:01 UTC) orbiter warning: context deadline exceeded Duration: 7.017993481s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [............X........X.......XX..X..XX..X.X.........XX.XX...] duration: 3.571407935s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:50:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571430558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X.....X.........X.X.....X.......X......X.......] duration: 8.858933923s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:50:32 UTC) orbiter warning: context deadline exceeded Duration: 8.858998114s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XX.X.........] duration: 14.827705143s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 14.827735401s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.............X........X.......XX..X..XX..X.X.........XX.XX..] duration: 3.616949642s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:49:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.617691603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X.....X.........X.X.....X.......X......X......] duration: 7.352838064s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:49:31 UTC) orbiter warning: context deadline exceeded Duration: 7.352895653s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X.....X.........X.X.....X.......X......X.....] duration: 26.906094969s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:49:13 UTC) orbiter warning: context deadline exceeded Duration: 26.906121139s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XX.X........] duration: 15.31531939s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:48:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 15.315349887s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X.....X.........X.X.....X.......X......X....] duration: 6.433490761s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:48:37 UTC) orbiter warning: context deadline exceeded Duration: 6.433728344s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..............X........X.......XX..X..XX..X.X.........XX.XX.] duration: 3.861024874s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:48:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.86105386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...............X........X.......XX..X..XX..X.X.........XX.XX] duration: 5.001579025s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:48:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001606075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XX.X.......] duration: 15.417566273s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:47:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 15.417586431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XX.X......] duration: 16.188130936s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:47:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 16.188161436s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X.....X.........X.X.....X.......X......X...] duration: 10.885314913s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:47:27 UTC) orbiter warning: context deadline exceeded Duration: 10.885342387s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.................X.....X.........X.X.....X.......X......X..] duration: 23.505482491s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:47:05 UTC) orbiter warning: context deadline exceeded Duration: 23.50551376s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [................X........X.......XX..X..XX..X.X.........XX.X] duration: 5.001570509s trace: https://jaeger.dev.storj.io/trace/76c6588ff4d02582 FAIL (2025-11-22 14:46:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001603681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XX.X.....] duration: 22.900700116s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:46:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 22.900732508s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.................X.....X.........X.X.....X.......X......X.] duration: 27.206250765s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:46:11 UTC) orbiter warning: context deadline exceeded Duration: 27.206278317s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.................X........X.......XX..X..XX..X.X.........XX.] duration: 4.171897962s trace: https://jaeger.dev.storj.io/trace/54280af9336f517e FAIL (2025-11-22 14:46:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.171946965s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)