Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 880302 reports (Page 11489 of 17607)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.365540456s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:34:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 3.365652219s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX.] duration: 3.570348413s trace: https://jaeger.dev.storj.io/trace/4a34f1d23c2208a FAIL (2025-12-16 15:34:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570373871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.945159069s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:34:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 3.945194907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..............X.X.............X............................] duration: 5.81052641s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:33:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.810559293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..............X.X.............X...........................] duration: 5.792901361s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:33:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.792945284s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX] duration: 5.001661512s trace: https://jaeger.dev.storj.io/trace/4a34f1d23c2208a FAIL (2025-12-16 15:33:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00169662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.792500656s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:33:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 2.792530534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..............X.X.............X..........................] duration: 5.741941234s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:33:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.741980698s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.401832242s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:32:49 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.401858883s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXX] duration: 5.000868872s trace: https://jaeger.dev.storj.io/trace/144e6bc337ab18b4 FAIL (2025-12-16 15:32:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000912746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.938454213s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:32:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 1.938484691s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.322175762s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:32:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.322213073s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.827750598s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:31:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 2.827781988s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..............X.X.............X.........................] duration: 5.712502188s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:31:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.712523398s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 804.414896ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:31:44 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 804.444281ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [..XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXX] duration: 5.001573604s trace: https://jaeger.dev.storj.io/trace/493a5d8d2b3ce550 FAIL (2025-12-16 15:31:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001592338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..............X.X.............X........................] duration: 5.652074701s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.652104589s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.414127224s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:31:25 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.414154276s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 14.59961887s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:31:00 UTC) orbiter warning: context deadline exceeded Duration: 14.599635712s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 7.26815032s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:30:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 7.268180508s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [...XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXX] duration: 5.001119219s trace: https://jaeger.dev.storj.io/trace/6edc5d52a91d95ee FAIL (2025-12-16 15:30:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001143607s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.422591591s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:30:33 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.422634684s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [....XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXX] duration: 5.000944957s trace: https://jaeger.dev.storj.io/trace/64e10c8d07e2fd96 FAIL (2025-12-16 15:30:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000978723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..............X.X.............X.......................] duration: 5.733968982s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:30:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.734001195s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.593138061s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:30:18 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.59316891s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 14.14651389s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:30:01 UTC) orbiter warning: context deadline exceeded Duration: 14.146545779s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.207168034s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:29:48 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 3.207197832s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.....XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XX] duration: 5.000981757s trace: https://jaeger.dev.storj.io/trace/1bd3db06271220b0 FAIL (2025-12-16 15:29:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001003447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X..............X.X.............X......................] duration: 6.084642826s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 6.084672411s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.431783475s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:29:08 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.43181263s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 17.683728411s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:28:58 UTC) orbiter warning: context deadline exceeded Duration: 17.683756204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X..............X.X.............X.....................] duration: 5.778236065s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:28:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.77824974s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.409329569s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:28:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 4.409360727s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 16.478263868s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:28:31 UTC) orbiter warning: context deadline exceeded Duration: 16.478304926s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [......XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..X] duration: 5.000890856s trace: https://jaeger.dev.storj.io/trace/53c6ca495a0287ac FAIL (2025-12-16 15:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000915763s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.083569914s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:28:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 6.083595101s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.507529658s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:28:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 1.507567201s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....X..............X.X.............X....................] duration: 6.009594532s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:27:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 6.009624179s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 12.115191129s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:27:44 UTC) orbiter warning: context deadline exceeded Duration: 12.115221435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.....X..............X.X.............X...................] duration: 5.841788721s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.841809881s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X......XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..] duration: 4.000542246s trace: https://jaeger.dev.storj.io/trace/35a7ab1d2e05b97d FAIL (2025-12-16 15:27:29 UTC) orbiter warning: context deadline exceeded Duration: 4.000600046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.104553042s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:27:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 4.10463612s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 905.271483ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:27:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 905.300699ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.....X..............X.X.............X..................] duration: 5.95987683s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:26:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.959895556s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 13.159712318s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:26:46 UTC) orbiter warning: context deadline exceeded Duration: 13.159756451s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX......XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX.] duration: 3.950349011s trace: https://jaeger.dev.storj.io/trace/35a7ab1d2e05b97d FAIL (2025-12-16 15:26:29 UTC) orbiter warning: context deadline exceeded Duration: 3.950379137s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.616206078s trace: https://jaeger.dev.storj.io/trace/3a1dc91c80dc63f3 FAIL (2025-12-16 15:26:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:54980->45.38.24.2:443: read: connection reset by peer Duration: 4.616335926s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 804.911872ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/11de0f622f04477a FAIL (2025-12-16 15:26:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": read tcp 188.166.225.170:51540->45.38.24.2:443: read: connection reset by peer Duration: 804.940125ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 8.652774846s trace: https://jaeger.dev.storj.io/trace/71c62388c8eb83a1 FAIL (2025-12-16 15:25:46 UTC) orbiter warning: context deadline exceeded Duration: 8.652803311s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXX......XX.XXXXXXX..X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX] duration: 5.001482932s trace: https://jaeger.dev.storj.io/trace/35a7ab1d2e05b97d FAIL (2025-12-16 15:25:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001505484s core uplink us-select-1 us1-placement-20

1
...
11487
11488
11489
11490
11491
...
17607