Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 900724 reports (Page 11328 of 18015)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X........X.............X..............................] duration: 19.6467593s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:39:09 UTC) orbiter warning: context deadline exceeded Duration: 19.646794218s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXX] duration: 5.001149545s trace: https://jaeger.dev.storj.io/trace/2ca6d168187e7645 FAIL (2025-12-28 14:39:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001173921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X...X....................X...........X.....] duration: 12.315603429s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:38:58 UTC) orbiter warning: context deadline exceeded Duration: 12.31564092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...X........X.............X.............................] duration: 15.832164404s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:38:47 UTC) orbiter warning: context deadline exceeded Duration: 15.832196393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...X...X] duration: 30.002410149s trace: https://jaeger.dev.storj.io/trace/e5ec8e7f5ef2aa8 FAIL (2025-12-28 14:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 14:38:15.795640238 (551h9m15.666723273s) [upload=483232]: Starting upload Dec 28 14:38:15.795708929 (551h9m15.666791964s) [upload=483232]: Done starting upload Dec 28 14:38:15.795743285 (551h9m15.66682632s) [upload=483232]: Uploading segments... Dec 28 14:38:15.795911806 (551h9m15.666994851s) [upload=483232]: Got next segment. Inline: false Dec 28 14:38:15.795958985 (551h9m15.66704202s) [upload=483232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:38:15.795963954 (551h9m15.667046999s) [upload=483232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:38:16.046023339 (551h9m15.917106374s) [seg_pos={0 0}, upload=483232]: Begin upload segment... Dec 28 14:38:16.046086660 (551h9m15.917169705s) [seg_pos={0 0}, upload=483232]: Done begin upload segment. Dec 28 14:38:16.046103061 (551h9m15.917186106s) [noise=true , address=77.90.31.15:4697 , piece_id=PAT4C7IGXLWW5OERUBP2CRHHIAEAXIS4CVTH6Z6PSPYJM4TEIA4A, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.046221016 (551h9m15.917304061s) [noise=true , address=45.91.116.11:4120 , piece_id=BIZWBXRRXPPFC6ZXZW6V4LGYTR4NTASRBYXXIN6JC76TDVCW5MZA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.046921129 (551h9m15.918004164s) [noise=true , address=77.90.31.17:4770 , piece_id=XU4O6IXL4UQ74373O5V7MD5FMN3BKL7J3TWLMQX4PGOAM5KZQNFQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.047304699 (551h9m15.918387734s) [noise=true , address=145.79.227.45:4229 , piece_id=IRWVFPUSYRP6XGTGTLNDFKOGI7XL56N2CEFLVSSMFWPAQ5BJALEQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.047701334 (551h9m15.918784379s) [noise=true , address=145.79.227.52:4445 , piece_id=KC5DQIA6YHU2LHNBMKVZIV43JEAJKY57SCXE4CHKOOPIHJHFIRWQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.048088271 (551h9m15.919171306s) [noise=true , address=145.79.227.12:4767 , piece_id=UAVN2NTMD2G4ZIZ3UFOTXFFVSBIPXJEIPLJJA43WKNTTT4D2QLVQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.048553768 (551h9m15.919636803s) [noise=true , address=145.79.227.4:4186 , piece_id=APTLKQA3KRF6FQPAOVMYZ3Q7LLEGO42UEKFINH4BZJ5NBQBH2ANA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.049005258 (551h9m15.920088303s) [noise=true , address=145.79.227.42:4123 , piece_id=JLCGS3LWD2NE5F66F2P74NKNMUONFDAOVKANJEK7JT3WLQA7KNWA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.050195103 (551h9m15.921278148s) [noise=true , address=145.79.227.18:4007 , piece_id=POXG5AJ5QNLVOUIXRYP46JU32TTKGWSQUDSZXPYTOZBD6URHEY3A, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.050215892 (551h9m15.921298937s) [noise=true , address=145.79.227.13:4804 , piece_id=E3ENA5HUM2O344GVYYFXWDAZ3LVHKJTZJ4OPZDPEWJGEMGC2UYDA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.051475351 (551h9m15.922558386s) [noise=true , address=145.79.227.50:4374 , piece_id=4TYE2KPDEOQ27XGWW2Z7B3WCUQQZ35OAP7YG7KFD74QSMFCF7ESQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.052704692 (551h9m15.923787737s) [noise=true , address=145.79.227.7:4294 , piece_id=UG7VHBA5KHQESIAAOFWENRKDOL42IB2HMEHZQ7HNFGQCX56D24EQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.053952498 (551h9m15.925035533s) [noise=true , address=77.90.31.16:4735 , piece_id=7KMD55E4HZGKZVEH5YB2HKN5TP2VNDYBGGBBAQSERHE36B5CBSQA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.055293020 (551h9m15.926376065s) [noise=true , address=77.90.31.14:4664 , piece_id=KWIHHXROECCSWZYPPPGESHZHRMGN6XODCMUZYSFRA5IPWWA357BA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.056601562 (551h9m15.927684597s) [noise=true , address=145.79.227.14:4839 , piece_id=B2LSSRCSDDQLK3QFDZTR6C3HLIM5XEBBF2DL6F2J5UMN7O2UVOEQ, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:16.057864487 (551h9m15.928947532s) [noise=true , address=145.79.227.6:4257 , piece_id=X3R5RYIW3DNPQISSG4MYLME2UHE72FTDYET5E3WOJKWGNK7EHWTA, seg_pos={0 0}, upload=483232]: Uploading piece... Dec 28 14:38:17.689295490 (551h9m17.560378525s) [upload=483232]: Next returned nil segment Dec 28 14:38:17.689308986 (551h9m17.560392021s) [upload=483232]: Waiting for error group managing segments... Dec 28 14:38:18.064334406 (551h9m17.935417441s) [noise=true , address=145.79.227.52:4445 , piece_id=KC5DQIA6YHU2LHNBMKVZIV43JEAJKY57SCXE4CHKOOPIHJHFIRWQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.065248367 (551h9m17.936331402s) [noise=true , address=145.79.227.14:4839 , piece_id=B2LSSRCSDDQLK3QFDZTR6C3HLIM5XEBBF2DL6F2J5UMN7O2UVOEQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.067583443 (551h9m17.938666488s) [noise=true , address=145.79.227.4:4186 , piece_id=APTLKQA3KRF6FQPAOVMYZ3Q7LLEGO42UEKFINH4BZJ5NBQBH2ANA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.071838044 (551h9m17.942921079s) [noise=true , address=145.79.227.12:4767 , piece_id=UAVN2NTMD2G4ZIZ3UFOTXFFVSBIPXJEIPLJJA43WKNTTT4D2QLVQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.072443147 (551h9m17.943526182s) [noise=true , address=145.79.227.6:4257 , piece_id=X3R5RYIW3DNPQISSG4MYLME2UHE72FTDYET5E3WOJKWGNK7EHWTA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.072939261 (551h9m17.944022306s) [noise=true , address=145.79.227.50:4374 , piece_id=4TYE2KPDEOQ27XGWW2Z7B3WCUQQZ35OAP7YG7KFD74QSMFCF7ESQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.085727542 (551h9m17.956810577s) [noise=true , address=145.79.227.18:4007 , piece_id=POXG5AJ5QNLVOUIXRYP46JU32TTKGWSQUDSZXPYTOZBD6URHEY3A, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.089789738 (551h9m17.960872773s) [noise=true , address=145.79.227.42:4123 , piece_id=JLCGS3LWD2NE5F66F2P74NKNMUONFDAOVKANJEK7JT3WLQA7KNWA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.090828034 (551h9m17.961911079s) [noise=true , address=145.79.227.13:4804 , piece_id=E3ENA5HUM2O344GVYYFXWDAZ3LVHKJTZJ4OPZDPEWJGEMGC2UYDA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.090852821 (551h9m17.961935856s) [noise=true , address=145.79.227.7:4294 , piece_id=UG7VHBA5KHQESIAAOFWENRKDOL42IB2HMEHZQ7HNFGQCX56D24EQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.091268042 (551h9m17.962351077s) [noise=true , address=145.79.227.45:4229 , piece_id=IRWVFPUSYRP6XGTGTLNDFKOGI7XL56N2CEFLVSSMFWPAQ5BJALEQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.229498636 (551h9m18.100581671s) [noise=true , address=77.90.31.16:4735 , piece_id=7KMD55E4HZGKZVEH5YB2HKN5TP2VNDYBGGBBAQSERHE36B5CBSQA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.312704674 (551h9m18.18378772s) [noise=true , address=77.90.31.17:4770 , piece_id=XU4O6IXL4UQ74373O5V7MD5FMN3BKL7J3TWLMQX4PGOAM5KZQNFQ, seg_pos={0 0}, upload=483232]: Done uploading piece. err: <nil> Dec 28 14:38:18.312726746 (551h9m18.183809781s) [seg_pos={0 0}, upload=483232]: Segment reached optimal threshold of 13 pieces. Dec 28 14:38:18.313102592 (551h9m18.184185667s) [noise=true , address=77.90.31.14:4664 , piece_id=KWIHHXROECCSWZYPPPGESHZHRMGN6XODCMUZYSFRA5IPWWA357BA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 14:38:18.313114615 (551h9m18.18419765s) [noise=true , address=77.90.31.14:4664 , piece_id=KWIHHXROECCSWZYPPPGESHZHRMGN6XODCMUZYSFRA5IPWWA357BA, seg_pos={0 0}, upload=483232]: Long tail cancellation due to optimal threshold Dec 28 14:38:18.313696533 (551h9m18.184779578s) [noise=true , address=45.91.116.11:4120 , piece_id=BIZWBXRRXPPFC6ZXZW6V4LGYTR4NTASRBYXXIN6JC76TDVCW5MZA, seg_pos={0 0}, upload=483232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 14:38:18.313706312 (551h9m18.184789357s) [noise=true , address=45.91.116.11:4120 , piece_id=BIZWBXRRXPPFC6ZXZW6V4LGYTR4NTASRBYXXIN6JC76TDVCW5MZA, seg_pos={0 0}, upload=483232]: Long tail cancellation due to optimal threshold Dec 28 14:38:18.313924046 (551h9m18.185007091s) [noise=true , address=77.90.31.15:4697 , piece_id=PAT4C7IGXLWW5OERUBP2CRHHIAEAXIS4CVTH6Z6PSPYJM4TEIA4A, seg_pos={0 0}, upload=483232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 14:38:18.313940688 (551h9m18.185023723s) [noise=true , address=77.90.31.15:4697 , piece_id=PAT4C7IGXLWW5OERUBP2CRHHIAEAXIS4CVTH6Z6PSPYJM4TEIA4A, seg_pos={0 0}, upload=483232]: Long tail cancellation due to optimal threshold Dec 28 14:38:18.313970424 (551h9m18.185053459s) [seg_pos={0 0}, upload=483232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:38:18.314059404 (551h9m18.185142439s) [upload=483232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:38:18.314072759 (551h9m18.185155814s) [upload=483232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:38:18.599209376 (551h9m18.470292422s) [upload=483232]: Done uploading segments... Dec 28 14:38:18.599234394 (551h9m18.470317439s) [upload=483232]: Upload finished. err: <nil> Duration: 30.002435878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X........X...X............X...............X....] duration: 42.313156109s trace: https://jaeger.dev.storj.io/trace/ccfaca51be04534 FAIL (2025-12-28 14:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=483168]: Uploading piece... Dec 28 14:33:54.320935734 (551h4m54.192018779s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321432371 (551h4m54.192515406s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321768721 (551h4m54.192851756s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323181501 (551h4m54.194264536s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323521168 (551h4m54.194604213s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323931459 (551h4m54.195014504s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324361228 (551h4m54.195444273s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324735951 (551h4m54.195818987s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324750228 (551h4m54.195833263s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325112718 (551h4m54.196195763s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325261522 (551h4m54.196344557s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325483424 (551h4m54.196566469s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325851155 (551h4m54.1969342s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325955794 (551h4m54.197038839s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.326737582 (551h4m54.197820617s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327283522 (551h4m54.198366567s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327876201 (551h4m54.198959236s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:55.653961030 (551h4m55.525044075s) [upload=483168]: Next returned nil segment Dec 28 14:33:55.653975428 (551h4m55.525058473s) [upload=483168]: Waiting for error group managing segments... Dec 28 14:33:55.991926802 (551h4m55.863009847s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.027096040 (551h4m55.898179075s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.164683799 (551h4m56.035766844s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.248292854 (551h4m56.119375889s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.382580940 (551h4m56.253663985s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.403355778 (551h4m56.274438814s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.482547292 (551h4m56.353630337s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.538380677 (551h4m56.409463722s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.104630578 (551h4m56.975713623s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.131673222 (551h4m57.002756268s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.196059755 (551h4m57.06714279s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.213134227 (551h4m57.084217262s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.249946403 (551h4m57.121029438s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253240687 (551h4m57.124323722s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253257298 (551h4m57.124340333s) [seg_pos={0 0}, upload=483168]: Segment reached optimal threshold of 14 pieces. Dec 28 14:33:57.253523295 (551h4m57.12460633s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253531029 (551h4m57.124614074s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253827264 (551h4m57.124910299s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253836190 (551h4m57.124919226s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253990494 (551h4m57.125073529s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253996857 (551h4m57.125079892s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254417508 (551h4m57.125500563s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.254426114 (551h4m57.125509159s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254445721 (551h4m57.125528756s) [seg_pos={0 0}, upload=483168]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:33:57.254516065 (551h4m57.1255991s) [upload=483168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:33:57.254520434 (551h4m57.125603469s) [upload=483168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:33:57.550161842 (551h4m57.421244877s) [upload=483168]: Done uploading segments... Dec 28 14:33:57.550178384 (551h4m57.421261429s) [upload=483168]: Upload finished. err: <nil> Duration: 42.313181727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [X..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXX] duration: 5.000871906s trace: https://jaeger.dev.storj.io/trace/7372564888897861 FAIL (2025-12-28 14:38:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000887065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X...X....................X...........X....] duration: 20.853485859s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:38:04 UTC) orbiter warning: context deadline exceeded Duration: 20.853517628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...X........X.............X............................] duration: 21.608792616s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:37:59 UTC) orbiter warning: context deadline exceeded Duration: 21.609069374s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X........X...X............X...............X...] duration: 53.947891732s trace: https://jaeger.dev.storj.io/trace/ccfaca51be04534 FAIL (2025-12-28 14:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=483168]: Uploading piece... Dec 28 14:33:54.320935734 (551h4m54.192018779s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321432371 (551h4m54.192515406s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321768721 (551h4m54.192851756s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323181501 (551h4m54.194264536s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323521168 (551h4m54.194604213s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323931459 (551h4m54.195014504s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324361228 (551h4m54.195444273s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324735951 (551h4m54.195818987s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324750228 (551h4m54.195833263s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325112718 (551h4m54.196195763s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325261522 (551h4m54.196344557s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325483424 (551h4m54.196566469s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325851155 (551h4m54.1969342s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325955794 (551h4m54.197038839s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.326737582 (551h4m54.197820617s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327283522 (551h4m54.198366567s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327876201 (551h4m54.198959236s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:55.653961030 (551h4m55.525044075s) [upload=483168]: Next returned nil segment Dec 28 14:33:55.653975428 (551h4m55.525058473s) [upload=483168]: Waiting for error group managing segments... Dec 28 14:33:55.991926802 (551h4m55.863009847s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.027096040 (551h4m55.898179075s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.164683799 (551h4m56.035766844s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.248292854 (551h4m56.119375889s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.382580940 (551h4m56.253663985s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.403355778 (551h4m56.274438814s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.482547292 (551h4m56.353630337s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.538380677 (551h4m56.409463722s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.104630578 (551h4m56.975713623s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.131673222 (551h4m57.002756268s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.196059755 (551h4m57.06714279s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.213134227 (551h4m57.084217262s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.249946403 (551h4m57.121029438s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253240687 (551h4m57.124323722s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253257298 (551h4m57.124340333s) [seg_pos={0 0}, upload=483168]: Segment reached optimal threshold of 14 pieces. Dec 28 14:33:57.253523295 (551h4m57.12460633s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253531029 (551h4m57.124614074s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253827264 (551h4m57.124910299s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253836190 (551h4m57.124919226s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253990494 (551h4m57.125073529s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253996857 (551h4m57.125079892s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254417508 (551h4m57.125500563s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.254426114 (551h4m57.125509159s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254445721 (551h4m57.125528756s) [seg_pos={0 0}, upload=483168]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:33:57.254516065 (551h4m57.1255991s) [upload=483168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:33:57.254520434 (551h4m57.125603469s) [upload=483168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:33:57.550161842 (551h4m57.421244877s) [upload=483168]: Done uploading segments... Dec 28 14:33:57.550178384 (551h4m57.421261429s) [upload=483168]: Upload finished. err: <nil> Duration: 53.947920787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXX] duration: 5.000806852s trace: https://jaeger.dev.storj.io/trace/34a3cb474ff0ec28 FAIL (2025-12-28 14:37:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000833381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...X........X.............X...........................] duration: 21.999968246s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:37:26 UTC) orbiter warning: context deadline exceeded Duration: 21.999999317s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X...X....................X...........X...] duration: 5.923169972s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:37:24 UTC) orbiter warning: context deadline exceeded Duration: 5.923200221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...X...] duration: 6.133060709s trace: https://jaeger.dev.storj.io/trace/4b8b77371c97072d FAIL (2025-12-28 14:37:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ) context deadline exceeded ======LOGS====== Dec 28 14:34:05.090957395 (551h5m4.96204043s) [upload=483172]: Starting upload Dec 28 14:34:05.091085348 (551h5m4.962168393s) [upload=483172]: Done starting upload Dec 28 14:34:05.091281983 (551h5m4.962365028s) [upload=483172]: Uploading segments... Dec 28 14:34:05.091509055 (551h5m4.96259209s) [upload=483172]: Got next segment. Inline: false Dec 28 14:34:05.091623163 (551h5m4.962706208s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:34:05.091632100 (551h5m4.962715145s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:34:05.340723699 (551h5m5.211806744s) [seg_pos={0 0}, upload=483172]: Begin upload segment... Dec 28 14:34:05.340790567 (551h5m5.211873612s) [seg_pos={0 0}, upload=483172]: Done begin upload segment. Dec 28 14:34:05.340830503 (551h5m5.211913548s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.340875058 (551h5m5.211958123s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341475922 (551h5m5.212558967s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341943532 (551h5m5.213026587s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342083539 (551h5m5.213166584s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342515342 (551h5m5.213598377s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342540860 (551h5m5.213623895s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342821545 (551h5m5.21390458s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342948816 (551h5m5.214031862s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343119101 (551h5m5.214202136s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343431766 (551h5m5.214514821s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343447637 (551h5m5.214530682s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343756275 (551h5m5.21483932s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343900199 (551h5m5.214983244s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344038021 (551h5m5.215121056s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344353312 (551h5m5.215436347s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:07.062159282 (551h5m6.933242327s) [upload=483172]: Next returned nil segment Dec 28 14:34:07.062185562 (551h5m6.933268597s) [upload=483172]: Waiting for error group managing segments... Dec 28 14:34:07.346940921 (551h5m7.218023966s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.348156456 (551h5m7.219239491s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.351534338 (551h5m7.222617383s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352131535 (551h5m7.22321457s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352708895 (551h5m7.22379194s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.357996694 (551h5m7.229079739s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.367592356 (551h5m7.238675401s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.369896253 (551h5m7.240979288s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.370584202 (551h5m7.241667237s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.372403396 (551h5m7.243486441s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.381578698 (551h5m7.252661743s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.516975254 (551h5m7.388058289s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736202017 (551h5m7.607285052s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736221535 (551h5m7.60730457s) [seg_pos={0 0}, upload=483172]: Segment reached optimal threshold of 13 pieces. Dec 28 14:34:07.736400586 (551h5m7.607483621s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736407689 (551h5m7.607490734s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736638759 (551h5m7.607721795s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736645692 (551h5m7.607728738s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736660380 (551h5m7.607743445s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection Dec 28 14:34:07.736681230 (551h5m7.607764265s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736743018 (551h5m7.607826053s) [seg_pos={0 0}, upload=483172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:34:07.736848399 (551h5m7.607931434s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:34:07.736855031 (551h5m7.607938076s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:34:08.032166853 (551h5m7.903249898s) [upload=483172]: Done uploading segments... Dec 28 14:34:08.032187061 (551h5m7.903270096s) [upload=483172]: Upload finished. err: <nil> Duration: 6.133192289s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX] duration: 5.000589316s trace: https://jaeger.dev.storj.io/trace/6b7ac2c784b6d1 FAIL (2025-12-28 14:37:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00061206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X........X...X............X...............X..] duration: 59.806293165s trace: https://jaeger.dev.storj.io/trace/ccfaca51be04534 FAIL (2025-12-28 14:36:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=483168]: Uploading piece... Dec 28 14:33:54.320935734 (551h4m54.192018779s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321432371 (551h4m54.192515406s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321768721 (551h4m54.192851756s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323181501 (551h4m54.194264536s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323521168 (551h4m54.194604213s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323931459 (551h4m54.195014504s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324361228 (551h4m54.195444273s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324735951 (551h4m54.195818987s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324750228 (551h4m54.195833263s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325112718 (551h4m54.196195763s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325261522 (551h4m54.196344557s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325483424 (551h4m54.196566469s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325851155 (551h4m54.1969342s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325955794 (551h4m54.197038839s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.326737582 (551h4m54.197820617s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327283522 (551h4m54.198366567s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327876201 (551h4m54.198959236s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:55.653961030 (551h4m55.525044075s) [upload=483168]: Next returned nil segment Dec 28 14:33:55.653975428 (551h4m55.525058473s) [upload=483168]: Waiting for error group managing segments... Dec 28 14:33:55.991926802 (551h4m55.863009847s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.027096040 (551h4m55.898179075s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.164683799 (551h4m56.035766844s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.248292854 (551h4m56.119375889s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.382580940 (551h4m56.253663985s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.403355778 (551h4m56.274438814s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.482547292 (551h4m56.353630337s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.538380677 (551h4m56.409463722s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.104630578 (551h4m56.975713623s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.131673222 (551h4m57.002756268s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.196059755 (551h4m57.06714279s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.213134227 (551h4m57.084217262s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.249946403 (551h4m57.121029438s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253240687 (551h4m57.124323722s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253257298 (551h4m57.124340333s) [seg_pos={0 0}, upload=483168]: Segment reached optimal threshold of 14 pieces. Dec 28 14:33:57.253523295 (551h4m57.12460633s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253531029 (551h4m57.124614074s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253827264 (551h4m57.124910299s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253836190 (551h4m57.124919226s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253990494 (551h4m57.125073529s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253996857 (551h4m57.125079892s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254417508 (551h4m57.125500563s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.254426114 (551h4m57.125509159s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254445721 (551h4m57.125528756s) [seg_pos={0 0}, upload=483168]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:33:57.254516065 (551h4m57.1255991s) [upload=483168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:33:57.254520434 (551h4m57.125603469s) [upload=483168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:33:57.550161842 (551h4m57.421244877s) [upload=483168]: Done uploading segments... Dec 28 14:33:57.550178384 (551h4m57.421261429s) [upload=483168]: Upload finished. err: <nil> Duration: 59.806317139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................X...X....................X...........X..] duration: 13.282497123s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:36:37 UTC) orbiter warning: context deadline exceeded Duration: 13.282523905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...X........X.............X..........................] duration: 27.431106492s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:36:17 UTC) orbiter warning: context deadline exceeded Duration: 27.431136359s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.X] duration: 5.001654426s trace: https://jaeger.dev.storj.io/trace/22f7c91134bcabfc FAIL (2025-12-28 14:36:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001670697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...X....................X...........X.] duration: 38.464632863s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:36:04 UTC) orbiter warning: context deadline exceeded Duration: 38.464657371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...X..] duration: 5.712632914s trace: https://jaeger.dev.storj.io/trace/4b8b77371c97072d FAIL (2025-12-28 14:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ) context deadline exceeded ======LOGS====== Dec 28 14:34:05.090957395 (551h5m4.96204043s) [upload=483172]: Starting upload Dec 28 14:34:05.091085348 (551h5m4.962168393s) [upload=483172]: Done starting upload Dec 28 14:34:05.091281983 (551h5m4.962365028s) [upload=483172]: Uploading segments... Dec 28 14:34:05.091509055 (551h5m4.96259209s) [upload=483172]: Got next segment. Inline: false Dec 28 14:34:05.091623163 (551h5m4.962706208s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:34:05.091632100 (551h5m4.962715145s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:34:05.340723699 (551h5m5.211806744s) [seg_pos={0 0}, upload=483172]: Begin upload segment... Dec 28 14:34:05.340790567 (551h5m5.211873612s) [seg_pos={0 0}, upload=483172]: Done begin upload segment. Dec 28 14:34:05.340830503 (551h5m5.211913548s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.340875058 (551h5m5.211958123s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341475922 (551h5m5.212558967s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341943532 (551h5m5.213026587s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342083539 (551h5m5.213166584s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342515342 (551h5m5.213598377s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342540860 (551h5m5.213623895s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342821545 (551h5m5.21390458s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342948816 (551h5m5.214031862s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343119101 (551h5m5.214202136s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343431766 (551h5m5.214514821s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343447637 (551h5m5.214530682s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343756275 (551h5m5.21483932s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343900199 (551h5m5.214983244s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344038021 (551h5m5.215121056s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344353312 (551h5m5.215436347s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:07.062159282 (551h5m6.933242327s) [upload=483172]: Next returned nil segment Dec 28 14:34:07.062185562 (551h5m6.933268597s) [upload=483172]: Waiting for error group managing segments... Dec 28 14:34:07.346940921 (551h5m7.218023966s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.348156456 (551h5m7.219239491s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.351534338 (551h5m7.222617383s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352131535 (551h5m7.22321457s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352708895 (551h5m7.22379194s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.357996694 (551h5m7.229079739s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.367592356 (551h5m7.238675401s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.369896253 (551h5m7.240979288s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.370584202 (551h5m7.241667237s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.372403396 (551h5m7.243486441s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.381578698 (551h5m7.252661743s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.516975254 (551h5m7.388058289s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736202017 (551h5m7.607285052s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736221535 (551h5m7.60730457s) [seg_pos={0 0}, upload=483172]: Segment reached optimal threshold of 13 pieces. Dec 28 14:34:07.736400586 (551h5m7.607483621s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736407689 (551h5m7.607490734s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736638759 (551h5m7.607721795s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736645692 (551h5m7.607728738s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736660380 (551h5m7.607743445s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection Dec 28 14:34:07.736681230 (551h5m7.607764265s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736743018 (551h5m7.607826053s) [seg_pos={0 0}, upload=483172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:34:07.736848399 (551h5m7.607931434s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:34:07.736855031 (551h5m7.607938076s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:34:08.032166853 (551h5m7.903249898s) [upload=483172]: Done uploading segments... Dec 28 14:34:08.032187061 (551h5m7.903270096s) [upload=483172]: Upload finished. err: <nil> Duration: 5.712670135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X........X...X............X...............X.] duration: 44.766130855s trace: https://jaeger.dev.storj.io/trace/ccfaca51be04534 FAIL (2025-12-28 14:35:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=483168]: Uploading piece... Dec 28 14:33:54.320935734 (551h4m54.192018779s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321432371 (551h4m54.192515406s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321768721 (551h4m54.192851756s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323181501 (551h4m54.194264536s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323521168 (551h4m54.194604213s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323931459 (551h4m54.195014504s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324361228 (551h4m54.195444273s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324735951 (551h4m54.195818987s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324750228 (551h4m54.195833263s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325112718 (551h4m54.196195763s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325261522 (551h4m54.196344557s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325483424 (551h4m54.196566469s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325851155 (551h4m54.1969342s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325955794 (551h4m54.197038839s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.326737582 (551h4m54.197820617s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327283522 (551h4m54.198366567s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327876201 (551h4m54.198959236s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:55.653961030 (551h4m55.525044075s) [upload=483168]: Next returned nil segment Dec 28 14:33:55.653975428 (551h4m55.525058473s) [upload=483168]: Waiting for error group managing segments... Dec 28 14:33:55.991926802 (551h4m55.863009847s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.027096040 (551h4m55.898179075s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.164683799 (551h4m56.035766844s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.248292854 (551h4m56.119375889s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.382580940 (551h4m56.253663985s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.403355778 (551h4m56.274438814s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.482547292 (551h4m56.353630337s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.538380677 (551h4m56.409463722s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.104630578 (551h4m56.975713623s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.131673222 (551h4m57.002756268s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.196059755 (551h4m57.06714279s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.213134227 (551h4m57.084217262s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.249946403 (551h4m57.121029438s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253240687 (551h4m57.124323722s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253257298 (551h4m57.124340333s) [seg_pos={0 0}, upload=483168]: Segment reached optimal threshold of 14 pieces. Dec 28 14:33:57.253523295 (551h4m57.12460633s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253531029 (551h4m57.124614074s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253827264 (551h4m57.124910299s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253836190 (551h4m57.124919226s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253990494 (551h4m57.125073529s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253996857 (551h4m57.125079892s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254417508 (551h4m57.125500563s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.254426114 (551h4m57.125509159s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254445721 (551h4m57.125528756s) [seg_pos={0 0}, upload=483168]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:33:57.254516065 (551h4m57.1255991s) [upload=483168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:33:57.254520434 (551h4m57.125603469s) [upload=483168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:33:57.550161842 (551h4m57.421244877s) [upload=483168]: Done uploading segments... Dec 28 14:33:57.550178384 (551h4m57.421261429s) [upload=483168]: Upload finished. err: <nil> Duration: 44.766146014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................X...X....................X...........X] duration: 1m0.000826863s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 14:35:25 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000852611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...X........X.............X.........................] duration: 25.863202656s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:35:07 UTC) orbiter warning: context deadline exceeded Duration: 25.863222394s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...X.] duration: 6.598172309s trace: https://jaeger.dev.storj.io/trace/4b8b77371c97072d FAIL (2025-12-28 14:35:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ) context deadline exceeded ======LOGS====== Dec 28 14:34:05.090957395 (551h5m4.96204043s) [upload=483172]: Starting upload Dec 28 14:34:05.091085348 (551h5m4.962168393s) [upload=483172]: Done starting upload Dec 28 14:34:05.091281983 (551h5m4.962365028s) [upload=483172]: Uploading segments... Dec 28 14:34:05.091509055 (551h5m4.96259209s) [upload=483172]: Got next segment. Inline: false Dec 28 14:34:05.091623163 (551h5m4.962706208s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:34:05.091632100 (551h5m4.962715145s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:34:05.340723699 (551h5m5.211806744s) [seg_pos={0 0}, upload=483172]: Begin upload segment... Dec 28 14:34:05.340790567 (551h5m5.211873612s) [seg_pos={0 0}, upload=483172]: Done begin upload segment. Dec 28 14:34:05.340830503 (551h5m5.211913548s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.340875058 (551h5m5.211958123s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341475922 (551h5m5.212558967s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341943532 (551h5m5.213026587s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342083539 (551h5m5.213166584s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342515342 (551h5m5.213598377s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342540860 (551h5m5.213623895s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342821545 (551h5m5.21390458s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342948816 (551h5m5.214031862s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343119101 (551h5m5.214202136s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343431766 (551h5m5.214514821s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343447637 (551h5m5.214530682s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343756275 (551h5m5.21483932s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343900199 (551h5m5.214983244s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344038021 (551h5m5.215121056s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344353312 (551h5m5.215436347s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:07.062159282 (551h5m6.933242327s) [upload=483172]: Next returned nil segment Dec 28 14:34:07.062185562 (551h5m6.933268597s) [upload=483172]: Waiting for error group managing segments... Dec 28 14:34:07.346940921 (551h5m7.218023966s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.348156456 (551h5m7.219239491s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.351534338 (551h5m7.222617383s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352131535 (551h5m7.22321457s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352708895 (551h5m7.22379194s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.357996694 (551h5m7.229079739s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.367592356 (551h5m7.238675401s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.369896253 (551h5m7.240979288s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.370584202 (551h5m7.241667237s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.372403396 (551h5m7.243486441s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.381578698 (551h5m7.252661743s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.516975254 (551h5m7.388058289s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736202017 (551h5m7.607285052s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736221535 (551h5m7.60730457s) [seg_pos={0 0}, upload=483172]: Segment reached optimal threshold of 13 pieces. Dec 28 14:34:07.736400586 (551h5m7.607483621s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736407689 (551h5m7.607490734s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736638759 (551h5m7.607721795s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736645692 (551h5m7.607728738s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736660380 (551h5m7.607743445s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection Dec 28 14:34:07.736681230 (551h5m7.607764265s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736743018 (551h5m7.607826053s) [seg_pos={0 0}, upload=483172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:34:07.736848399 (551h5m7.607931434s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:34:07.736855031 (551h5m7.607938076s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:34:08.032166853 (551h5m7.903249898s) [upload=483172]: Done uploading segments... Dec 28 14:34:08.032187061 (551h5m7.903270096s) [upload=483172]: Upload finished. err: <nil> Duration: 6.598204882s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.] duration: 3.619504439s trace: https://jaeger.dev.storj.io/trace/55f29a2ac6bbf398 FAIL (2025-12-28 14:34:56 UTC) orbiter warning: context deadline exceeded Duration: 3.619533033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X........X...X............X...............X] duration: 1m0.00264292s trace: https://jaeger.dev.storj.io/trace/ccfaca51be04534 FAIL (2025-12-28 14:34:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=483168]: Uploading piece... Dec 28 14:33:54.320935734 (551h4m54.192018779s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321432371 (551h4m54.192515406s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.321768721 (551h4m54.192851756s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323181501 (551h4m54.194264536s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323521168 (551h4m54.194604213s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.323931459 (551h4m54.195014504s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324361228 (551h4m54.195444273s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324735951 (551h4m54.195818987s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.324750228 (551h4m54.195833263s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325112718 (551h4m54.196195763s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325261522 (551h4m54.196344557s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325483424 (551h4m54.196566469s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325851155 (551h4m54.1969342s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.325955794 (551h4m54.197038839s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.326737582 (551h4m54.197820617s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327283522 (551h4m54.198366567s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:54.327876201 (551h4m54.198959236s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Uploading piece... Dec 28 14:33:55.653961030 (551h4m55.525044075s) [upload=483168]: Next returned nil segment Dec 28 14:33:55.653975428 (551h4m55.525058473s) [upload=483168]: Waiting for error group managing segments... Dec 28 14:33:55.991926802 (551h4m55.863009847s) [noise=true , address=64.49.13.154:4927 , piece_id=EURBDR2PH4RKSNZF44YAAMVLBOVRTSKTQ7RCG2OD3EOAUZBZCHHQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.027096040 (551h4m55.898179075s) [noise=true , address=64.49.13.170:4718 , piece_id=C62LPOZGS3FJZXP3VJGFWT3MSH7NWDWJHSRQSJQQB6U7BWB4Y55A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.164683799 (551h4m56.035766844s) [noise=true , address=193.149.164.143:4699 , piece_id=O2WSZNQY4J3YDDLCWNGCKXNG2QFWSPHLWT3TL5TPPVN5ULFLZVEA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.248292854 (551h4m56.119375889s) [noise=true , address=145.79.227.35:4490 , piece_id=TCLUXK3BS3P5FBDKFQJ4WKBXDCR4EEYN4F4U2ZFFE5LNWX3QBYAA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.382580940 (551h4m56.253663985s) [noise=true , address=77.90.31.17:4778 , piece_id=V3TQWDCGCVUFOZMLSE2UWWVKF3GIDFTLNDRN5AXEMSTEJBZFSOUQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.403355778 (551h4m56.274438814s) [noise=true , address=77.90.31.16:4740 , piece_id=5KUZZWFTGLNUS6FW3IC5ZZFGKK3K7CTDOWWXD3EM4NGUBZ2IUBKA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.482547292 (551h4m56.353630337s) [noise=true , address=77.90.31.13:4630 , piece_id=R3FMXFP7ZXDOXLPWOBF373LJOTI53YBTQJVPUGQX7XIWOTH42SWQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:56.538380677 (551h4m56.409463722s) [noise=true , address=45.91.116.14:4218 , piece_id=PIG57MRIEUMYDSSM3D5SCAD2M5Q467AUNZLMF7K3BEXLYNSRQ3IQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.104630578 (551h4m56.975713623s) [noise=true , address=145.79.227.44:4208 , piece_id=SL742MK3H5CA6VP2E2ZUEPRQFTFJGFR6NWCNT7B4NZWLYTDWGXRA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.131673222 (551h4m57.002756268s) [noise=true , address=145.79.227.30:4794 , piece_id=C7P2ZDJAP2I22D2FZ7NBDC6ZPI4TDA6G3WZT3EKQAWH7RNCKXDPA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.196059755 (551h4m57.06714279s) [noise=true , address=145.79.227.34:4925 , piece_id=DQRZTRFZPJECH56K7VSSHIGTF5VMOCKFHTDPNVOXH5EIJ33BVH5A, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.213134227 (551h4m57.084217262s) [noise=true , address=145.79.227.49:4372 , piece_id=A2W5ALJYY5YOCHKZCEYAECPO2IZTTHUGZPHRJA2SW3E7HE3Z5KYA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.249946403 (551h4m57.121029438s) [noise=true , address=77.90.31.18:4839 , piece_id=HEEFAEKZBYLWGXPCYVDLYPROQD4BQW6BHKE6N6FHFS4TDFUFG4FQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253240687 (551h4m57.124323722s) [noise=true , address=45.91.116.12:4232 , piece_id=LSUX3366W666QUQA47IVSITJNAUFLO5GW2HFUVRYXOCAD6T3JCSA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: <nil> Dec 28 14:33:57.253257298 (551h4m57.124340333s) [seg_pos={0 0}, upload=483168]: Segment reached optimal threshold of 14 pieces. Dec 28 14:33:57.253523295 (551h4m57.12460633s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253531029 (551h4m57.124614074s) [noise=true , address=77.90.31.15:4705 , piece_id=5STWSW5SMTY3RZLUTRCA7EOSDWWSQVETG4Z5VIKKO5DAYT22ITCA, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253827264 (551h4m57.124910299s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253836190 (551h4m57.124919226s) [noise=true , address=145.79.227.40:4054 , piece_id=3G4ESEZBKE66KTJKDAOX3WXLD6ENWLPY2Y7BBAOPFUE5633OW57Q, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.253990494 (551h4m57.125073529s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.253996857 (551h4m57.125079892s) [noise=true , address=77.90.31.12:4608 , piece_id=BQVJRKKBBWR53EBTHC3MLZBWE4SAH4GVUVXGZ7DW2YCESRUIUEXQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254417508 (551h4m57.125500563s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 28 14:33:57.254426114 (551h4m57.125509159s) [noise=true , address=77.90.31.21:4889 , piece_id=T6FOQVIYR4GEQRIQYTNJPH6TQXVHIXZYZAT46HSBQ6VTVGLUVANQ, seg_pos={0 0}, upload=483168]: Long tail cancellation due to optimal threshold Dec 28 14:33:57.254445721 (551h4m57.125528756s) [seg_pos={0 0}, upload=483168]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:33:57.254516065 (551h4m57.1255991s) [upload=483168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:33:57.254520434 (551h4m57.125603469s) [upload=483168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:33:57.550161842 (551h4m57.421244877s) [upload=483168]: Done uploading segments... Dec 28 14:33:57.550178384 (551h4m57.421261429s) [upload=483168]: Upload finished. err: <nil> Duration: 1m0.002668959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...X........X.............X........................] duration: 24.395111826s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:34:41 UTC) orbiter warning: context deadline exceeded Duration: 24.395141321s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...X] duration: 30.004239767s trace: https://jaeger.dev.storj.io/trace/4b8b77371c97072d FAIL (2025-12-28 14:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ) context deadline exceeded ======LOGS====== Dec 28 14:34:05.090957395 (551h5m4.96204043s) [upload=483172]: Starting upload Dec 28 14:34:05.091085348 (551h5m4.962168393s) [upload=483172]: Done starting upload Dec 28 14:34:05.091281983 (551h5m4.962365028s) [upload=483172]: Uploading segments... Dec 28 14:34:05.091509055 (551h5m4.96259209s) [upload=483172]: Got next segment. Inline: false Dec 28 14:34:05.091623163 (551h5m4.962706208s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:34:05.091632100 (551h5m4.962715145s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:34:05.340723699 (551h5m5.211806744s) [seg_pos={0 0}, upload=483172]: Begin upload segment... Dec 28 14:34:05.340790567 (551h5m5.211873612s) [seg_pos={0 0}, upload=483172]: Done begin upload segment. Dec 28 14:34:05.340830503 (551h5m5.211913548s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.340875058 (551h5m5.211958123s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341475922 (551h5m5.212558967s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.341943532 (551h5m5.213026587s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342083539 (551h5m5.213166584s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342515342 (551h5m5.213598377s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342540860 (551h5m5.213623895s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342821545 (551h5m5.21390458s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.342948816 (551h5m5.214031862s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343119101 (551h5m5.214202136s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343431766 (551h5m5.214514821s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343447637 (551h5m5.214530682s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343756275 (551h5m5.21483932s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.343900199 (551h5m5.214983244s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344038021 (551h5m5.215121056s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:05.344353312 (551h5m5.215436347s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Uploading piece... Dec 28 14:34:07.062159282 (551h5m6.933242327s) [upload=483172]: Next returned nil segment Dec 28 14:34:07.062185562 (551h5m6.933268597s) [upload=483172]: Waiting for error group managing segments... Dec 28 14:34:07.346940921 (551h5m7.218023966s) [noise=true , address=145.79.227.12:4765 , piece_id=XGKRITN7BZZK4TH6LRBCK2Z6R2XSZC6XAPJWA5YXVM75W2634FPQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.348156456 (551h5m7.219239491s) [noise=true , address=145.79.227.51:4410 , piece_id=ULJ4XFGDOQZW2NWU5EI4D32T6ZPRYH4RNVEDT7MTP6ZC5ZAHCJUQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.351534338 (551h5m7.222617383s) [noise=true , address=145.79.227.52:4445 , piece_id=AK7N62XVICXZ2VZMSBHCV47KQEIR5FXG3QZGE6FL44GV4K7AHUKQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352131535 (551h5m7.22321457s) [noise=true , address=145.79.227.3:4150 , piece_id=67DJAAQBRDL3VMOVPWWAQLXVYEFR7DYTSPROLS4WWHLQIGNTUJYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.352708895 (551h5m7.22379194s) [noise=true , address=145.79.227.8:4330 , piece_id=TP74VHIDENGIYYAPLNDFD2B5RA7YQ5YZI6DGDYWNDTEUJJ3M3Z3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.357996694 (551h5m7.229079739s) [noise=true , address=145.79.227.13:4802 , piece_id=K3TLM2HLFUNWUZ7HOQANYJ4UQQR2RHWEX6Q7DXAPGNYOISCC6MBA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.367592356 (551h5m7.238675401s) [noise=true , address=145.79.227.45:4231 , piece_id=3UY3ACLKDT3PZ56UVB4KL2S3M3KN4YUIC4PW5GROGCI6NLB7F63Q, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.369896253 (551h5m7.240979288s) [noise=true , address=145.79.227.6:4259 , piece_id=LXXXDKY6VRDNS4JHZ2VMRWG2O6NS3RSSPY3G6UXIECC33K6IONSA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.370584202 (551h5m7.241667237s) [noise=true , address=145.79.227.46:4601 , piece_id=HHIMAHD2MOAAC2JZ6W2XN63WIQLBA5NPJFE2P4KVM7WZ376IBYXA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.372403396 (551h5m7.243486441s) [noise=true , address=145.79.227.16:4909 , piece_id=YXIVOIAXH3JQXMCCIYRKMFWMGP3QPRJFWXDLJJSQHHWXAY4PCB3A, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.381578698 (551h5m7.252661743s) [noise=true , address=145.79.227.25:4619 , piece_id=OVM47Y3QHQKFGJCH4UB2DYQ4T3RBBWMPZXFDZSRO37BJZA566PCA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.516975254 (551h5m7.388058289s) [noise=true , address=77.90.31.10:4519 , piece_id=3FFV43LQRXZF6VSPHMQHX2CLPUB34M6XPUK4MCSVRYUI2F56IKFA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736202017 (551h5m7.607285052s) [noise=true , address=45.91.116.6:4975 , piece_id=5AP2EHOAUSB3Z65CNMMW6AK5ALS32FWE3V3WWUB75MHXIYT4NHJQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: <nil> Dec 28 14:34:07.736221535 (551h5m7.60730457s) [seg_pos={0 0}, upload=483172]: Segment reached optimal threshold of 13 pieces. Dec 28 14:34:07.736400586 (551h5m7.607483621s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736407689 (551h5m7.607490734s) [noise=true , address=45.91.116.14:4189 , piece_id=2FALVUH5PBTY5FPVBFY3A2TL2RPMZOPWUAZF724XZQ5LC4HNNPMQ, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736638759 (551h5m7.607721795s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 14:34:07.736645692 (551h5m7.607728738s) [noise=true , address=45.91.116.9:4262 , piece_id=QFQLQWGHI7B6KNZX6KDD6BY5PQKPGYXHDCALI6726HUZNOUGWEMA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736660380 (551h5m7.607743445s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57108->45.91.116.8:4046: use of closed network connection Dec 28 14:34:07.736681230 (551h5m7.607764265s) [noise=true , address=45.91.116.8:4046 , piece_id=KUH76RWBKL5HAQAFJQWQDHBCWU5BUSARB67SRPXGPDUIVVT5OVYA, seg_pos={0 0}, upload=483172]: Long tail cancellation due to optimal threshold Dec 28 14:34:07.736743018 (551h5m7.607826053s) [seg_pos={0 0}, upload=483172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:34:07.736848399 (551h5m7.607931434s) [upload=483172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:34:07.736855031 (551h5m7.607938076s) [upload=483172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:34:08.032166853 (551h5m7.903249898s) [upload=483172]: Done uploading segments... Dec 28 14:34:08.032187061 (551h5m7.903270096s) [upload=483172]: Upload finished. err: <nil> Duration: 30.004267759s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX] duration: 5.00163462s trace: https://jaeger.dev.storj.io/trace/55f29a2ac6bbf398 FAIL (2025-12-28 14:34:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001654789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...X....................X...........] duration: 10.034119141s trace: https://jaeger.dev.storj.io/trace/155e9c46ca29887d FAIL (2025-12-28 14:34:04 UTC) orbiter warning: context deadline exceeded Duration: 10.034142346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...X........X.............X.......................] duration: 19.494857525s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:34:02 UTC) orbiter warning: context deadline exceeded Duration: 19.494892752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X........X...X............X...............] duration: 43.362854774s trace: https://jaeger.dev.storj.io/trace/6e3d0e082ccfd6df FAIL (2025-12-28 14:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891251117 (550h51m27.762334163s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891414809 (550h51m27.762497854s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891784283 (550h51m27.762867318s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891817427 (550h51m27.762900462s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892162783 (550h51m27.763245818s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892387462 (550h51m27.763470507s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892512410 (550h51m27.763595455s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892853268 (550h51m27.763936313s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892947999 (550h51m27.764031034s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893186894 (550h51m27.764269939s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893470474 (550h51m27.764553519s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893494570 (550h51m27.764577615s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894238977 (550h51m27.765322022s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894787081 (550h51m27.765870126s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.895167355 (550h51m27.7662504s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:29.358894118 (550h51m29.229977164s) [upload=482965]: Next returned nil segment Dec 28 14:20:29.358907694 (550h51m29.229990739s) [upload=482965]: Waiting for error group managing segments... Dec 28 14:20:29.765028634 (550h51m29.636111669s) [noise=true , address=145.79.227.11:4746 , piece_id=S677GW3O2RN7QCTMBEAEI2J7EON77MSXUEKPEPCSD4H33GG6LI4Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.766370650 (550h51m29.637453695s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.773707891 (550h51m29.644790936s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.790413170 (550h51m29.661496205s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.937355352 (550h51m29.808438397s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.941527726 (550h51m29.812610761s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.975364798 (550h51m29.846447833s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.036770315 (550h51m29.90785336s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.081766965 (550h51m29.95285001s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.400645452 (550h51m30.271728497s) [noise=true , address=64.49.13.137:4495 , piece_id=EZU7KF5W55D7EK5FPJHGGU52TBUAHFLARI6UO42XJZLGGIWFJQMQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.612950915 (550h51m30.48403395s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.655758848 (550h51m30.526841893s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.669703671 (550h51m30.540786716s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703943981 (550h51m30.575027016s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703969599 (550h51m30.575052644s) [seg_pos={0 0}, upload=482965]: Segment reached optimal threshold of 14 pieces. Dec 28 14:20:30.704385441 (550h51m30.575468486s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection Dec 28 14:20:30.704396472 (550h51m30.575479507s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704491342 (550h51m30.575574387s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704540867 (550h51m30.575623912s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704613155 (550h51m30.5756962s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704622122 (550h51m30.575705167s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704867148 (550h51m30.575950193s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704876295 (550h51m30.57595934s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.705082157 (550h51m30.576165202s) [seg_pos={0 0}, upload=482965]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:20:30.705243795 (550h51m30.576326851s) [upload=482965]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:20:30.705251720 (550h51m30.576334765s) [upload=482965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:20:31.003436773 (550h51m30.874519818s) [upload=482965]: Done uploading segments... Dec 28 14:20:31.003458895 (550h51m30.87454195s) [upload=482965]: Upload finished. err: <nil> Duration: 43.36290012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X...X........X.............X......................] duration: 24.163713631s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:33:29 UTC) orbiter warning: context deadline exceeded Duration: 24.163738328s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X...X....................X..........] duration: 11.906672539s trace: https://jaeger.dev.storj.io/trace/155e9c46ca29887d FAIL (2025-12-28 14:33:12 UTC) orbiter warning: context deadline exceeded Duration: 11.90669939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.X] duration: 5.001023766s trace: https://jaeger.dev.storj.io/trace/37eb58c10da73ac3 FAIL (2025-12-28 14:33:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001043935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X........X...X............X..............] duration: 47.854793294s trace: https://jaeger.dev.storj.io/trace/6e3d0e082ccfd6df FAIL (2025-12-28 14:33:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891251117 (550h51m27.762334163s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891414809 (550h51m27.762497854s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891784283 (550h51m27.762867318s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891817427 (550h51m27.762900462s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892162783 (550h51m27.763245818s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892387462 (550h51m27.763470507s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892512410 (550h51m27.763595455s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892853268 (550h51m27.763936313s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892947999 (550h51m27.764031034s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893186894 (550h51m27.764269939s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893470474 (550h51m27.764553519s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893494570 (550h51m27.764577615s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894238977 (550h51m27.765322022s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894787081 (550h51m27.765870126s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.895167355 (550h51m27.7662504s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:29.358894118 (550h51m29.229977164s) [upload=482965]: Next returned nil segment Dec 28 14:20:29.358907694 (550h51m29.229990739s) [upload=482965]: Waiting for error group managing segments... Dec 28 14:20:29.765028634 (550h51m29.636111669s) [noise=true , address=145.79.227.11:4746 , piece_id=S677GW3O2RN7QCTMBEAEI2J7EON77MSXUEKPEPCSD4H33GG6LI4Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.766370650 (550h51m29.637453695s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.773707891 (550h51m29.644790936s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.790413170 (550h51m29.661496205s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.937355352 (550h51m29.808438397s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.941527726 (550h51m29.812610761s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.975364798 (550h51m29.846447833s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.036770315 (550h51m29.90785336s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.081766965 (550h51m29.95285001s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.400645452 (550h51m30.271728497s) [noise=true , address=64.49.13.137:4495 , piece_id=EZU7KF5W55D7EK5FPJHGGU52TBUAHFLARI6UO42XJZLGGIWFJQMQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.612950915 (550h51m30.48403395s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.655758848 (550h51m30.526841893s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.669703671 (550h51m30.540786716s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703943981 (550h51m30.575027016s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703969599 (550h51m30.575052644s) [seg_pos={0 0}, upload=482965]: Segment reached optimal threshold of 14 pieces. Dec 28 14:20:30.704385441 (550h51m30.575468486s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection Dec 28 14:20:30.704396472 (550h51m30.575479507s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704491342 (550h51m30.575574387s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704540867 (550h51m30.575623912s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704613155 (550h51m30.5756962s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704622122 (550h51m30.575705167s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704867148 (550h51m30.575950193s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704876295 (550h51m30.57595934s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.705082157 (550h51m30.576165202s) [seg_pos={0 0}, upload=482965]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:20:30.705243795 (550h51m30.576326851s) [upload=482965]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:20:30.705251720 (550h51m30.576334765s) [upload=482965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:20:31.003436773 (550h51m30.874519818s) [upload=482965]: Done uploading segments... Dec 28 14:20:31.003458895 (550h51m30.87454195s) [upload=482965]: Upload finished. err: <nil> Duration: 47.854815156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X...] duration: 5.98699304s trace: https://jaeger.dev.storj.io/trace/218570ce1d794871 FAIL (2025-12-28 14:32:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA) context deadline exceeded ======LOGS====== Dec 28 14:30:26.037720963 (551h1m25.908804009s) [upload=483114]: Starting upload Dec 28 14:30:26.037880668 (551h1m25.908963713s) [upload=483114]: Done starting upload Dec 28 14:30:26.038049720 (551h1m25.909132755s) [upload=483114]: Uploading segments... Dec 28 14:30:26.038599267 (551h1m25.909682312s) [upload=483114]: Got next segment. Inline: false Dec 28 14:30:26.038645685 (551h1m25.90972872s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:30:26.038653810 (551h1m25.909736845s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:30:26.290087411 (551h1m26.161170476s) [seg_pos={0 0}, upload=483114]: Begin upload segment... Dec 28 14:30:26.290150151 (551h1m26.161233196s) [seg_pos={0 0}, upload=483114]: Done begin upload segment. Dec 28 14:30:26.290310476 (551h1m26.161393521s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.290645945 (551h1m26.161729s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291895324 (551h1m26.162978379s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291973613 (551h1m26.163056658s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292286148 (551h1m26.163369193s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292376079 (551h1m26.163459124s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292665059 (551h1m26.163748094s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292672964 (551h1m26.163755999s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293011018 (551h1m26.164094063s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293027971 (551h1m26.164111006s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293382436 (551h1m26.164465481s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293465453 (551h1m26.164548498s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294253394 (551h1m26.165336429s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294619360 (551h1m26.165702406s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295068756 (551h1m26.166151821s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295565974 (551h1m26.166649019s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:28.005171238 (551h1m27.876254293s) [upload=483114]: Next returned nil segment Dec 28 14:30:28.005185957 (551h1m27.876269002s) [upload=483114]: Waiting for error group managing segments... Dec 28 14:30:28.284146973 (551h1m28.155230008s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.291103900 (551h1m28.162186945s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.297967790 (551h1m28.169050835s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.299413482 (551h1m28.170496517s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.307442291 (551h1m28.178525336s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.308891750 (551h1m28.179974785s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.310841132 (551h1m28.181924177s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.318068936 (551h1m28.189151981s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.328690300 (551h1m28.199773336s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.350327521 (551h1m28.221410566s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.559320067 (551h1m28.430403102s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.661987582 (551h1m28.533070618s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672431780 (551h1m28.543514825s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672457860 (551h1m28.543540915s) [seg_pos={0 0}, upload=483114]: Segment reached optimal threshold of 13 pieces. Dec 28 14:30:28.672687817 (551h1m28.543770862s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.672696514 (551h1m28.543779559s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673041661 (551h1m28.544124706s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection Dec 28 14:30:28.673050659 (551h1m28.544133704s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673210142 (551h1m28.544293187s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.673218377 (551h1m28.544301432s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673237194 (551h1m28.544320239s) [seg_pos={0 0}, upload=483114]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:30:28.673319951 (551h1m28.544402996s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:30:28.673342203 (551h1m28.544425239s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:30:29.014924411 (551h1m28.886007456s) [upload=483114]: Done uploading segments... Dec 28 14:30:29.014943798 (551h1m28.886026843s) [upload=483114]: Upload finished. err: <nil> Duration: 5.987018309s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X...X........X.............X.....................] duration: 19.026470566s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:32:52 UTC) orbiter warning: context deadline exceeded Duration: 19.026496056s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...X....................X.........] duration: 25.608927614s trace: https://jaeger.dev.storj.io/trace/155e9c46ca29887d FAIL (2025-12-28 14:32:44 UTC) orbiter warning: context deadline exceeded Duration: 25.608953683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 3.50549597s trace: https://jaeger.dev.storj.io/trace/1f55aacafb497485 FAIL (2025-12-28 14:32:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505525356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X........X...X............X.............] duration: 40.850515609s trace: https://jaeger.dev.storj.io/trace/6e3d0e082ccfd6df FAIL (2025-12-28 14:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891251117 (550h51m27.762334163s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891414809 (550h51m27.762497854s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891784283 (550h51m27.762867318s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891817427 (550h51m27.762900462s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892162783 (550h51m27.763245818s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892387462 (550h51m27.763470507s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892512410 (550h51m27.763595455s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892853268 (550h51m27.763936313s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892947999 (550h51m27.764031034s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893186894 (550h51m27.764269939s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893470474 (550h51m27.764553519s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893494570 (550h51m27.764577615s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894238977 (550h51m27.765322022s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894787081 (550h51m27.765870126s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.895167355 (550h51m27.7662504s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:29.358894118 (550h51m29.229977164s) [upload=482965]: Next returned nil segment Dec 28 14:20:29.358907694 (550h51m29.229990739s) [upload=482965]: Waiting for error group managing segments... Dec 28 14:20:29.765028634 (550h51m29.636111669s) [noise=true , address=145.79.227.11:4746 , piece_id=S677GW3O2RN7QCTMBEAEI2J7EON77MSXUEKPEPCSD4H33GG6LI4Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.766370650 (550h51m29.637453695s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.773707891 (550h51m29.644790936s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.790413170 (550h51m29.661496205s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.937355352 (550h51m29.808438397s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.941527726 (550h51m29.812610761s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.975364798 (550h51m29.846447833s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.036770315 (550h51m29.90785336s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.081766965 (550h51m29.95285001s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.400645452 (550h51m30.271728497s) [noise=true , address=64.49.13.137:4495 , piece_id=EZU7KF5W55D7EK5FPJHGGU52TBUAHFLARI6UO42XJZLGGIWFJQMQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.612950915 (550h51m30.48403395s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.655758848 (550h51m30.526841893s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.669703671 (550h51m30.540786716s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703943981 (550h51m30.575027016s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703969599 (550h51m30.575052644s) [seg_pos={0 0}, upload=482965]: Segment reached optimal threshold of 14 pieces. Dec 28 14:20:30.704385441 (550h51m30.575468486s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection Dec 28 14:20:30.704396472 (550h51m30.575479507s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704491342 (550h51m30.575574387s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704540867 (550h51m30.575623912s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704613155 (550h51m30.5756962s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704622122 (550h51m30.575705167s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704867148 (550h51m30.575950193s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704876295 (550h51m30.57595934s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.705082157 (550h51m30.576165202s) [seg_pos={0 0}, upload=482965]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:20:30.705243795 (550h51m30.576326851s) [upload=482965]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:20:30.705251720 (550h51m30.576334765s) [upload=482965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:20:31.003436773 (550h51m30.874519818s) [upload=482965]: Done uploading segments... Dec 28 14:20:31.003458895 (550h51m30.87454195s) [upload=482965]: Upload finished. err: <nil> Duration: 40.850542018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X..] duration: 6.709720604s trace: https://jaeger.dev.storj.io/trace/218570ce1d794871 FAIL (2025-12-28 14:31:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA) context deadline exceeded ======LOGS====== Dec 28 14:30:26.037720963 (551h1m25.908804009s) [upload=483114]: Starting upload Dec 28 14:30:26.037880668 (551h1m25.908963713s) [upload=483114]: Done starting upload Dec 28 14:30:26.038049720 (551h1m25.909132755s) [upload=483114]: Uploading segments... Dec 28 14:30:26.038599267 (551h1m25.909682312s) [upload=483114]: Got next segment. Inline: false Dec 28 14:30:26.038645685 (551h1m25.90972872s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:30:26.038653810 (551h1m25.909736845s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:30:26.290087411 (551h1m26.161170476s) [seg_pos={0 0}, upload=483114]: Begin upload segment... Dec 28 14:30:26.290150151 (551h1m26.161233196s) [seg_pos={0 0}, upload=483114]: Done begin upload segment. Dec 28 14:30:26.290310476 (551h1m26.161393521s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.290645945 (551h1m26.161729s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291895324 (551h1m26.162978379s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291973613 (551h1m26.163056658s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292286148 (551h1m26.163369193s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292376079 (551h1m26.163459124s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292665059 (551h1m26.163748094s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292672964 (551h1m26.163755999s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293011018 (551h1m26.164094063s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293027971 (551h1m26.164111006s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293382436 (551h1m26.164465481s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293465453 (551h1m26.164548498s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294253394 (551h1m26.165336429s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294619360 (551h1m26.165702406s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295068756 (551h1m26.166151821s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295565974 (551h1m26.166649019s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:28.005171238 (551h1m27.876254293s) [upload=483114]: Next returned nil segment Dec 28 14:30:28.005185957 (551h1m27.876269002s) [upload=483114]: Waiting for error group managing segments... Dec 28 14:30:28.284146973 (551h1m28.155230008s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.291103900 (551h1m28.162186945s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.297967790 (551h1m28.169050835s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.299413482 (551h1m28.170496517s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.307442291 (551h1m28.178525336s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.308891750 (551h1m28.179974785s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.310841132 (551h1m28.181924177s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.318068936 (551h1m28.189151981s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.328690300 (551h1m28.199773336s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.350327521 (551h1m28.221410566s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.559320067 (551h1m28.430403102s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.661987582 (551h1m28.533070618s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672431780 (551h1m28.543514825s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672457860 (551h1m28.543540915s) [seg_pos={0 0}, upload=483114]: Segment reached optimal threshold of 13 pieces. Dec 28 14:30:28.672687817 (551h1m28.543770862s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.672696514 (551h1m28.543779559s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673041661 (551h1m28.544124706s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection Dec 28 14:30:28.673050659 (551h1m28.544133704s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673210142 (551h1m28.544293187s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.673218377 (551h1m28.544301432s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673237194 (551h1m28.544320239s) [seg_pos={0 0}, upload=483114]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:30:28.673319951 (551h1m28.544402996s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:30:28.673342203 (551h1m28.544425239s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:30:29.014924411 (551h1m28.886007456s) [upload=483114]: Done uploading segments... Dec 28 14:30:29.014943798 (551h1m28.886026843s) [upload=483114]: Upload finished. err: <nil> Duration: 6.709750331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X...X........X.............X....................] duration: 11.057589917s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:31:57 UTC) orbiter warning: context deadline exceeded Duration: 11.057611007s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.000639245s trace: https://jaeger.dev.storj.io/trace/1f55aacafb497485 FAIL (2025-12-28 14:31:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000663622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X...X........X.............X...................] duration: 18.648165288s trace: https://jaeger.dev.storj.io/trace/35a5d4335e2cde85 FAIL (2025-12-28 14:31:35 UTC) orbiter warning: context deadline exceeded Duration: 18.64819243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X........X...X............X............] duration: 53.990607628s trace: https://jaeger.dev.storj.io/trace/6e3d0e082ccfd6df FAIL (2025-12-28 14:31:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891251117 (550h51m27.762334163s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891414809 (550h51m27.762497854s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891784283 (550h51m27.762867318s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.891817427 (550h51m27.762900462s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892162783 (550h51m27.763245818s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892387462 (550h51m27.763470507s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892512410 (550h51m27.763595455s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892853268 (550h51m27.763936313s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.892947999 (550h51m27.764031034s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893186894 (550h51m27.764269939s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893470474 (550h51m27.764553519s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.893494570 (550h51m27.764577615s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894238977 (550h51m27.765322022s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.894787081 (550h51m27.765870126s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:27.895167355 (550h51m27.7662504s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Uploading piece... Dec 28 14:20:29.358894118 (550h51m29.229977164s) [upload=482965]: Next returned nil segment Dec 28 14:20:29.358907694 (550h51m29.229990739s) [upload=482965]: Waiting for error group managing segments... Dec 28 14:20:29.765028634 (550h51m29.636111669s) [noise=true , address=145.79.227.11:4746 , piece_id=S677GW3O2RN7QCTMBEAEI2J7EON77MSXUEKPEPCSD4H33GG6LI4Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.766370650 (550h51m29.637453695s) [noise=true , address=145.79.227.17:4981 , piece_id=LFOLMTY2LIB45ARMSVPUGSZBOQR72IMTVOHPFA5EUCUMK56KRXOQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.773707891 (550h51m29.644790936s) [noise=true , address=145.79.227.23:4564 , piece_id=LJDYIHKF4TRJ35DHE2M7G2FCWYLXCSWF74FIYBLLGSTH2C64Q46Q, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.790413170 (550h51m29.661496205s) [noise=true , address=145.79.227.7:4316 , piece_id=7VUBNJWH5JWOANMRVO2BRJF5JYEZV5ZLSGEH44N7NYRXBMZBRXVQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.937355352 (550h51m29.808438397s) [noise=true , address=77.90.31.11:4567 , piece_id=YQZEUUUSH2HSTKJZSV56YZLXVZC7MTTHHHE3LU5B6FHXE7J34DPA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.941527726 (550h51m29.812610761s) [noise=true , address=77.90.31.15:4728 , piece_id=OG2G25ILF5VVMCRSXL6E2VWB6GVCCLYMTHZE4YRPBVU2NI2LVUGQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:29.975364798 (550h51m29.846447833s) [noise=true , address=77.90.31.18:4815 , piece_id=T53JP4LUEPBX5YQIBSC4QUMOS2CFAFZZX5RTPZESTD5RUWSBI7LQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.036770315 (550h51m29.90785336s) [noise=true , address=77.90.31.19:4849 , piece_id=QJBMVYVWNZDEK2ZR6FEDB6BL7VHO5MXQULSJNUQSJB3MRLUGGSHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.081766965 (550h51m29.95285001s) [noise=true , address=77.90.31.12:4624 , piece_id=XFEKSZBK7MTGT5S5N3OE2HACFITCJ6YCNUGLMXNF37NX7JNJW6CA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.400645452 (550h51m30.271728497s) [noise=true , address=64.49.13.137:4495 , piece_id=EZU7KF5W55D7EK5FPJHGGU52TBUAHFLARI6UO42XJZLGGIWFJQMQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.612950915 (550h51m30.48403395s) [noise=true , address=77.90.31.13:4641 , piece_id=EWOLF6IZQ2ZEO6TRWAB7ML6N7EZO735E23G6NALADJX6OBU2ZIVA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.655758848 (550h51m30.526841893s) [noise=true , address=145.79.227.30:4765 , piece_id=BT4NLERERQQ23Q5WU56U6GE52TJYKH64DCHA32AQ3LCNSXUWDNPQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.669703671 (550h51m30.540786716s) [noise=true , address=145.79.227.44:4196 , piece_id=B6JKFGTY6DG5GO7I34O5UMXYWJUNBUXFPXHUIUE22G7UY6ZC44HQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703943981 (550h51m30.575027016s) [noise=true , address=145.79.227.34:4939 , piece_id=BNHRXY6XZJ3SHSAAJZFGO6OYKHMV2UZ56FHILGXH2HQVQ7VTVTUA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: <nil> Dec 28 14:20:30.703969599 (550h51m30.575052644s) [seg_pos={0 0}, upload=482965]: Segment reached optimal threshold of 14 pieces. Dec 28 14:20:30.704385441 (550h51m30.575468486s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56774->64.49.13.169:4710: use of closed network connection Dec 28 14:20:30.704396472 (550h51m30.575479507s) [noise=true , address=64.49.13.169:4710 , piece_id=IYNK4DDJQEUAXNIPGYH2HKQHR2BAACEXTUHXEE4XR2HNZCVSJDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704491342 (550h51m30.575574387s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704540867 (550h51m30.575623912s) [noise=true , address=77.90.31.16:4743 , piece_id=HXJN5LJMA4LZN5ZC3CMKIYN2I5WXFZGMFUOWU5ENMJUP4L52NPGA, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704613155 (550h51m30.5756962s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704622122 (550h51m30.575705167s) [noise=true , address=145.79.227.33:4896 , piece_id=6ZTHDX2NYFWB75GJPLPIPXSJB7M72ULM2KLA3SHX6D5H7VONFDHQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.704867148 (550h51m30.575950193s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9): protocol: expected piece hash; context canceled; EOF Dec 28 14:20:30.704876295 (550h51m30.57595934s) [noise=true , address=145.79.227.40:4084 , piece_id=KI4FGH547ZDSTUMKR552TNIA4AYR5KIMHF4RZJKWSM47J6BH7IIQ, seg_pos={0 0}, upload=482965]: Long tail cancellation due to optimal threshold Dec 28 14:20:30.705082157 (550h51m30.576165202s) [seg_pos={0 0}, upload=482965]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 14:20:30.705243795 (550h51m30.576326851s) [upload=482965]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:20:30.705251720 (550h51m30.576334765s) [upload=482965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:20:31.003436773 (550h51m30.874519818s) [upload=482965]: Done uploading segments... Dec 28 14:20:31.003458895 (550h51m30.87454195s) [upload=482965]: Upload finished. err: <nil> Duration: 53.990623889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...X....................X........] duration: 13.607092819s trace: https://jaeger.dev.storj.io/trace/155e9c46ca29887d FAIL (2025-12-28 14:31:18 UTC) orbiter warning: context deadline exceeded Duration: 13.607121796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X.] duration: 5.867663608s trace: https://jaeger.dev.storj.io/trace/218570ce1d794871 FAIL (2025-12-28 14:31:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA) context deadline exceeded ======LOGS====== Dec 28 14:30:26.037720963 (551h1m25.908804009s) [upload=483114]: Starting upload Dec 28 14:30:26.037880668 (551h1m25.908963713s) [upload=483114]: Done starting upload Dec 28 14:30:26.038049720 (551h1m25.909132755s) [upload=483114]: Uploading segments... Dec 28 14:30:26.038599267 (551h1m25.909682312s) [upload=483114]: Got next segment. Inline: false Dec 28 14:30:26.038645685 (551h1m25.90972872s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:30:26.038653810 (551h1m25.909736845s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:30:26.290087411 (551h1m26.161170476s) [seg_pos={0 0}, upload=483114]: Begin upload segment... Dec 28 14:30:26.290150151 (551h1m26.161233196s) [seg_pos={0 0}, upload=483114]: Done begin upload segment. Dec 28 14:30:26.290310476 (551h1m26.161393521s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.290645945 (551h1m26.161729s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291895324 (551h1m26.162978379s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291973613 (551h1m26.163056658s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292286148 (551h1m26.163369193s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292376079 (551h1m26.163459124s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292665059 (551h1m26.163748094s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292672964 (551h1m26.163755999s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293011018 (551h1m26.164094063s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293027971 (551h1m26.164111006s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293382436 (551h1m26.164465481s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293465453 (551h1m26.164548498s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294253394 (551h1m26.165336429s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294619360 (551h1m26.165702406s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295068756 (551h1m26.166151821s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295565974 (551h1m26.166649019s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:28.005171238 (551h1m27.876254293s) [upload=483114]: Next returned nil segment Dec 28 14:30:28.005185957 (551h1m27.876269002s) [upload=483114]: Waiting for error group managing segments... Dec 28 14:30:28.284146973 (551h1m28.155230008s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.291103900 (551h1m28.162186945s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.297967790 (551h1m28.169050835s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.299413482 (551h1m28.170496517s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.307442291 (551h1m28.178525336s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.308891750 (551h1m28.179974785s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.310841132 (551h1m28.181924177s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.318068936 (551h1m28.189151981s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.328690300 (551h1m28.199773336s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.350327521 (551h1m28.221410566s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.559320067 (551h1m28.430403102s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.661987582 (551h1m28.533070618s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672431780 (551h1m28.543514825s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672457860 (551h1m28.543540915s) [seg_pos={0 0}, upload=483114]: Segment reached optimal threshold of 13 pieces. Dec 28 14:30:28.672687817 (551h1m28.543770862s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.672696514 (551h1m28.543779559s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673041661 (551h1m28.544124706s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection Dec 28 14:30:28.673050659 (551h1m28.544133704s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673210142 (551h1m28.544293187s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.673218377 (551h1m28.544301432s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673237194 (551h1m28.544320239s) [seg_pos={0 0}, upload=483114]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:30:28.673319951 (551h1m28.544402996s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:30:28.673342203 (551h1m28.544425239s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:30:29.014924411 (551h1m28.886007456s) [upload=483114]: Done uploading segments... Dec 28 14:30:29.014943798 (551h1m28.886026843s) [upload=483114]: Upload finished. err: <nil> Duration: 5.867731146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X......XXX.XX....XX......XX.XX.X.X..XXXXX....X...X..X..X] duration: 30.003265154s trace: https://jaeger.dev.storj.io/trace/218570ce1d794871 FAIL (2025-12-28 14:30:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA) context deadline exceeded ======LOGS====== Dec 28 14:30:26.037720963 (551h1m25.908804009s) [upload=483114]: Starting upload Dec 28 14:30:26.037880668 (551h1m25.908963713s) [upload=483114]: Done starting upload Dec 28 14:30:26.038049720 (551h1m25.909132755s) [upload=483114]: Uploading segments... Dec 28 14:30:26.038599267 (551h1m25.909682312s) [upload=483114]: Got next segment. Inline: false Dec 28 14:30:26.038645685 (551h1m25.90972872s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 14:30:26.038653810 (551h1m25.909736845s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 14:30:26.290087411 (551h1m26.161170476s) [seg_pos={0 0}, upload=483114]: Begin upload segment... Dec 28 14:30:26.290150151 (551h1m26.161233196s) [seg_pos={0 0}, upload=483114]: Done begin upload segment. Dec 28 14:30:26.290310476 (551h1m26.161393521s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.290645945 (551h1m26.161729s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291895324 (551h1m26.162978379s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.291973613 (551h1m26.163056658s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292286148 (551h1m26.163369193s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292376079 (551h1m26.163459124s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292665059 (551h1m26.163748094s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.292672964 (551h1m26.163755999s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293011018 (551h1m26.164094063s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293027971 (551h1m26.164111006s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293382436 (551h1m26.164465481s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.293465453 (551h1m26.164548498s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294253394 (551h1m26.165336429s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.294619360 (551h1m26.165702406s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295068756 (551h1m26.166151821s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:26.295565974 (551h1m26.166649019s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Uploading piece... Dec 28 14:30:28.005171238 (551h1m27.876254293s) [upload=483114]: Next returned nil segment Dec 28 14:30:28.005185957 (551h1m27.876269002s) [upload=483114]: Waiting for error group managing segments... Dec 28 14:30:28.284146973 (551h1m28.155230008s) [noise=true , address=145.79.227.46:4603 , piece_id=PZ6IR74MHG2CXULFE2G45YE4VBCHJRBT7GPX2GQ5NVI37MLNYRRA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.291103900 (551h1m28.162186945s) [noise=true , address=145.79.227.52:4446 , piece_id=VNI2DPWFPCPYAFSWJIOCQH5HTCS7KYXX5ZCBTAQBIHUGSKXRCPQQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.297967790 (551h1m28.169050835s) [noise=true , address=145.79.227.51:4410 , piece_id=VJP3NGMTWGEMXFORXORF4IASLW4YPIMMQNXWIIORRFMXFU7H3VXA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.299413482 (551h1m28.170496517s) [noise=true , address=145.79.227.50:4373 , piece_id=TD7KZKCKZEV5ITRACNLYRFTDPLVF3P6LUHDN6H3WIDUDONQODZGQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.307442291 (551h1m28.178525336s) [noise=true , address=145.79.227.12:4767 , piece_id=XAACLBZ5CO36MMEVNSOOTHPKUS2VTRNGVHXKC63ONCUX6LCVCRQA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.308891750 (551h1m28.179974785s) [noise=true , address=145.79.227.48:4303 , piece_id=ZHLPHCHUXFAHRF55WIR7XT4ULRMRPIUSOKN6HZS4XTJ747YU63OQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.310841132 (551h1m28.181924177s) [noise=true , address=145.79.227.5:4221 , piece_id=5ZR5FBMJGNTA6LCQIOTPSMB3IQUWZBHBIPVORWHTDKCP56M3KOPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.318068936 (551h1m28.189151981s) [noise=true , address=145.79.227.47:4265 , piece_id=MH3HHU6JNBNAZJUHL73RFMRG7LSD4NWJ5ZNOROIZ3O3BM6ELSHPQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.328690300 (551h1m28.199773336s) [noise=true , address=145.79.227.36:4519 , piece_id=GJPTB2XIXW5AA4W5S72PHBICKIDFVAR5MQ7OVPQC74VNBVQU2BTQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.350327521 (551h1m28.221410566s) [noise=true , address=145.79.227.14:4839 , piece_id=HDX3X2SUYAWWOTQU5RSSHO4VJ2V6SJ46DBNYBE6TZZ56QAWABUEA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.559320067 (551h1m28.430403102s) [noise=true , address=45.91.116.10:4083 , piece_id=B7K4EBXPME2TWIWMAYRL2RHQ4V6UJVU4QAJ6H3V6Q7TO62A6BLKQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.661987582 (551h1m28.533070618s) [noise=true , address=45.91.116.7:4011 , piece_id=6OLSDHUQROHHI4WW2T5RQMVF5CB7S63BTY3ES3VQ67YBNZO6GETA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672431780 (551h1m28.543514825s) [noise=true , address=45.91.116.13:4153 , piece_id=IUZHHWEHVPOJ57IWBUO3KQD472A5YIU3TXN5VOERKLGOMU2YZOVA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: <nil> Dec 28 14:30:28.672457860 (551h1m28.543540915s) [seg_pos={0 0}, upload=483114]: Segment reached optimal threshold of 13 pieces. Dec 28 14:30:28.672687817 (551h1m28.543770862s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.672696514 (551h1m28.543779559s) [noise=true , address=77.90.31.19:4844 , piece_id=SWH2MBAVE4T6F2KFNCF2S4WQNDTOHXUK4ILE34EGMBMFPN4XC5SA, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673041661 (551h1m28.544124706s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49654->45.91.116.8:4046: use of closed network connection Dec 28 14:30:28.673050659 (551h1m28.544133704s) [noise=true , address=45.91.116.8:4046 , piece_id=D6CQBX6CWXMSPJ5UJVX7VJQDTIOIQQ6ZGQOVQ4ZJRJSEDEZE32WQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673210142 (551h1m28.544293187s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 14:30:28.673218377 (551h1m28.544301432s) [noise=true , address=77.90.31.21:4878 , piece_id=IST36QMK3L7F4GYDS7OBJW3DFC4DKHVONSGNQSA475IKAHN7ALAQ, seg_pos={0 0}, upload=483114]: Long tail cancellation due to optimal threshold Dec 28 14:30:28.673237194 (551h1m28.544320239s) [seg_pos={0 0}, upload=483114]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 14:30:28.673319951 (551h1m28.544402996s) [upload=483114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 14:30:28.673342203 (551h1m28.544425239s) [upload=483114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 14:30:29.014924411 (551h1m28.886007456s) [upload=483114]: Done uploading segments... Dec 28 14:30:29.014943798 (551h1m28.886026843s) [upload=483114]: Upload finished. err: <nil> Duration: 30.00328881s core uplink us-select-1

1
...
11326
11327
11328
11329
11330
...
18015