Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 904035 reports (Page 11285 of 18081)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X] duration: 30.003335734s trace: https://jaeger.dev.storj.io/trace/702e1a3c40e071fa FAIL (2025-12-29 15:03:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 15:02:50.701227468 (575h33m50.572310503s) [upload=504556]: Starting upload Dec 29 15:02:50.701363046 (575h33m50.572446091s) [upload=504556]: Done starting upload Dec 29 15:02:50.701618793 (575h33m50.572701838s) [upload=504556]: Uploading segments... Dec 29 15:02:50.701668668 (575h33m50.572751714s) [upload=504556]: Got next segment. Inline: false Dec 29 15:02:50.701701010 (575h33m50.572784045s) [upload=504556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 15:02:50.701706220 (575h33m50.572789255s) [upload=504556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 15:02:50.947509747 (575h33m50.818592792s) [seg_pos={0 0}, upload=504556]: Begin upload segment... Dec 29 15:02:50.947563980 (575h33m50.818647015s) [seg_pos={0 0}, upload=504556]: Done begin upload segment. Dec 29 15:02:50.947597634 (575h33m50.818680679s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.947640305 (575h33m50.81872334s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.947744273 (575h33m50.818827318s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.948295112 (575h33m50.819378157s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.948724810 (575h33m50.819807855s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.949131945 (575h33m50.820214991s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.949644953 (575h33m50.820727998s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.950078087 (575h33m50.821161122s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.950504049 (575h33m50.821587084s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.952658110 (575h33m50.823741155s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.954399415 (575h33m50.82548246s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.956064576 (575h33m50.827147611s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.957699199 (575h33m50.828782234s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.959484719 (575h33m50.830567764s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.961283905 (575h33m50.83236695s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:50.962191904 (575h33m50.833274949s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Uploading piece... Dec 29 15:02:52.646677396 (575h33m52.517760441s) [upload=504556]: Next returned nil segment Dec 29 15:02:52.646693147 (575h33m52.517776182s) [upload=504556]: Waiting for error group managing segments... Dec 29 15:02:52.929771059 (575h33m52.800854094s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.941900515 (575h33m52.81298356s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.962515018 (575h33m52.833598054s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.962797476 (575h33m52.833880521s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.971366052 (575h33m52.842449107s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.974205118 (575h33m52.845288163s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:52.974942312 (575h33m52.846025347s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.172408857 (575h33m53.043491902s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.230215170 (575h33m53.101298215s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.247996648 (575h33m53.119079693s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.269960580 (575h33m53.141043625s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.333047026 (575h33m53.204130061s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.337777604 (575h33m53.208860639s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil> Dec 29 15:02:53.337823802 (575h33m53.208906847s) [seg_pos={0 0}, upload=504556]: Segment reached optimal threshold of 13 pieces. Dec 29 15:02:53.338100459 (575h33m53.209183504s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 15:02:53.338109606 (575h33m53.209192641s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold Dec 29 15:02:53.338210368 (575h33m53.209293403s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 15:02:53.338225267 (575h33m53.209308302s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold Dec 29 15:02:53.338376965 (575h33m53.20946s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 29 15:02:53.338383087 (575h33m53.209466132s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold Dec 29 15:02:53.338416771 (575h33m53.209499816s) [seg_pos={0 0}, upload=504556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 15:02:53.338518154 (575h33m53.209601199s) [upload=504556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 15:02:53.338527612 (575h33m53.209610647s) [upload=504556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 15:02:53.626254487 (575h33m53.497337542s) [upload=504556]: Done uploading segments... Dec 29 15:02:53.626281308 (575h33m53.497364353s) [upload=504556]: Upload finished. err: <nil> Duration: 30.003361002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.131385098s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 15:03:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.131411719s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X........X.........X......X.......X...X....] duration: 50.730141645s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 15:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 50.730167063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................XXX...........XX.X..................] duration: 16.229793381s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 15:03:06 UTC) orbiter warning: context deadline exceeded Duration: 16.229809021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.............] duration: 15.755611377s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:02:53 UTC) orbiter warning: context deadline exceeded Duration: 15.755637327s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................XXX...........XX.X.................] duration: 41.687917806s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 15:02:50 UTC) orbiter warning: context deadline exceeded Duration: 41.687943807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001563046s trace: https://jaeger.dev.storj.io/trace/3dd429f8e224f764 FAIL (2025-12-29 15:02:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001582994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX............] duration: 29.581360668s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:02:17 UTC) orbiter warning: context deadline exceeded Duration: 29.58138785s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.000741945s trace: https://jaeger.dev.storj.io/trace/56bf4a2253997ff9 FAIL (2025-12-29 15:02:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000768174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X........X.........X......X.......X...X...] duration: 29.535955822s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 15:02:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 29.535982733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.747485496s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 15:02:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.747547274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.] duration: 6.14150244s trace: https://jaeger.dev.storj.io/trace/578f42818ce4b775 FAIL (2025-12-29 15:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA) context deadline exceeded ======LOGS====== Dec 29 15:00:46.230687400 (575h31m46.101770435s) [upload=504524]: Starting upload Dec 29 15:00:46.230745841 (575h31m46.101828886s) [upload=504524]: Done starting upload Dec 29 15:00:46.230852825 (575h31m46.10193586s) [upload=504524]: Uploading segments... Dec 29 15:00:46.230934129 (575h31m46.102017175s) [upload=504524]: Got next segment. Inline: false Dec 29 15:00:46.230993082 (575h31m46.102076117s) [upload=504524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 15:00:46.230999965 (575h31m46.10208301s) [upload=504524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 15:00:46.479113660 (575h31m46.350196705s) [seg_pos={0 0}, upload=504524]: Begin upload segment... Dec 29 15:00:46.479195305 (575h31m46.35027834s) [seg_pos={0 0}, upload=504524]: Done begin upload segment. Dec 29 15:00:46.479249448 (575h31m46.350332493s) [noise=true , address=45.91.116.9:4261 , piece_id=H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.479398883 (575h31m46.350481928s) [noise=true , address=77.90.31.12:4589 , piece_id=QPMAO6MQLJTOIUUGWYXHN7O6T3N6CKYMBMMK4LNTRCNI3GGKOW5A, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480061124 (575h31m46.351144169s) [noise=true , address=45.91.116.6:4976 , piece_id=6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480658030 (575h31m46.351741075s) [noise=true , address=45.91.116.7:4010 , piece_id=KMCWLJVCSTOZVIXSAM55C37VUATH6FW4O4HU7GUBQ6XSTBB56A2A, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480835388 (575h31m46.351918423s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480968511 (575h31m46.352051556s) [noise=true , address=145.79.227.37:4567 , piece_id=LP5CCC2EJ32EWOJS272O2NX6DRQEHF7MXRUZY5IE5NCGNM6JFKJA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481156961 (575h31m46.352239996s) [noise=true , address=145.79.227.5:4223 , piece_id=6FM54KX3AAHCWBMSEKXNDXEWKR5F6BPXGLSVZLRX56T2TZ7M3LWA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481228026 (575h31m46.352311061s) [noise=true , address=145.79.227.24:4582 , piece_id=KLSCJDC3CKBTMNIJLDPUMX5MGYOBFE2LXVHF2NLUCZKILTOY76IQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481611737 (575h31m46.352694772s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481679526 (575h31m46.352762571s) [noise=true , address=77.90.31.10:4520 , piece_id=SP64YJGCQHZ7ODA5RLZMIFFQKZ54Z6WBTVEHN5INI6LJXZSTRSJQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482031055 (575h31m46.3531141s) [noise=true , address=45.91.116.14:4190 , piece_id=QDCAIZRYZNHDY3FZTWFELYLCRYHH2ZG2H2HNVPK4N5OSP2B5LYXQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482060171 (575h31m46.353143206s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482417301 (575h31m46.353500336s) [noise=true , address=145.79.227.8:4331 , piece_id=4Y352FONHTEJ7D2NX6FPJV2I3643LK4JL7HAEVVGJTTYQADGCHGQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482439292 (575h31m46.353522327s) [noise=true , address=145.79.227.27:4690 , piece_id=S54XALCWQ7MXEH7H36EWJZ7KJPCY4E7VYVNFTWDYYLVOBJZS7DSQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482758561 (575h31m46.353841616s) [noise=true , address=145.79.227.15:4876 , piece_id=XSFNXDXL7WF45IQVNKI6Z6HOA643JZUOXFRKH6MUEVVQX4TWTRRQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.483191895 (575h31m46.35427493s) [noise=true , address=45.91.116.13:4154 , piece_id=E6IGXDVCKQJ6REFODY4R3CWVVLXC6MKRBPXIOEIXVNN7ACUF44BQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:48.094503725 (575h31m47.96558677s) [upload=504524]: Next returned nil segment Dec 29 15:00:48.094520918 (575h31m47.965603963s) [upload=504524]: Waiting for error group managing segments... Dec 29 15:00:48.473958274 (575h31m48.345041309s) [noise=true , address=145.79.227.5:4223 , piece_id=6FM54KX3AAHCWBMSEKXNDXEWKR5F6BPXGLSVZLRX56T2TZ7M3LWA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.476468684 (575h31m48.347551729s) [noise=true , address=145.79.227.37:4567 , piece_id=LP5CCC2EJ32EWOJS272O2NX6DRQEHF7MXRUZY5IE5NCGNM6JFKJA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.483300322 (575h31m48.354383357s) [noise=true , address=145.79.227.8:4331 , piece_id=4Y352FONHTEJ7D2NX6FPJV2I3643LK4JL7HAEVVGJTTYQADGCHGQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.484038157 (575h31m48.355121192s) [noise=true , address=145.79.227.27:4690 , piece_id=S54XALCWQ7MXEH7H36EWJZ7KJPCY4E7VYVNFTWDYYLVOBJZS7DSQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.487580062 (575h31m48.358663097s) [noise=true , address=145.79.227.24:4582 , piece_id=KLSCJDC3CKBTMNIJLDPUMX5MGYOBFE2LXVHF2NLUCZKILTOY76IQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.490870848 (575h31m48.361953883s) [noise=true , address=145.79.227.15:4876 , piece_id=XSFNXDXL7WF45IQVNKI6Z6HOA643JZUOXFRKH6MUEVVQX4TWTRRQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.698180386 (575h31m48.569263421s) [noise=true , address=45.91.116.14:4190 , piece_id=QDCAIZRYZNHDY3FZTWFELYLCRYHH2ZG2H2HNVPK4N5OSP2B5LYXQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.707969877 (575h31m48.579052922s) [noise=true , address=45.91.116.6:4976 , piece_id=6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.749966284 (575h31m48.621049319s) [noise=true , address=45.91.116.7:4010 , piece_id=KMCWLJVCSTOZVIXSAM55C37VUATH6FW4O4HU7GUBQ6XSTBB56A2A, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.765112295 (575h31m48.63619533s) [noise=true , address=77.90.31.10:4520 , piece_id=SP64YJGCQHZ7ODA5RLZMIFFQKZ54Z6WBTVEHN5INI6LJXZSTRSJQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.814885480 (575h31m48.685968525s) [noise=true , address=45.91.116.9:4261 , piece_id=H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.496478315 (575h31m49.36756136s) [noise=true , address=77.90.31.12:4589 , piece_id=QPMAO6MQLJTOIUUGWYXHN7O6T3N6CKYMBMMK4LNTRCNI3GGKOW5A, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.609434281 (575h31m49.480517336s) [noise=true , address=45.91.116.13:4154 , piece_id=E6IGXDVCKQJ6REFODY4R3CWVVLXC6MKRBPXIOEIXVNN7ACUF44BQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.609461562 (575h31m49.480544607s) [seg_pos={0 0}, upload=504524]: Segment reached optimal threshold of 13 pieces. Dec 29 15:00:49.609704565 (575h31m49.48078763s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.609714214 (575h31m49.480797259s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.609731626 (575h31m49.480814702s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.609743099 (575h31m49.480826164s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.610040595 (575h31m49.48112365s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.610050223 (575h31m49.481133268s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.610076473 (575h31m49.481159508s) [seg_pos={0 0}, upload=504524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 15:00:49.610189077 (575h31m49.481272122s) [upload=504524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 15:00:49.610197915 (575h31m49.48128096s) [upload=504524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 15:00:49.906765905 (575h31m49.77784894s) [upload=504524]: Done uploading segments... Dec 29 15:00:49.906785301 (575h31m49.777868337s) [upload=504524]: Upload finished. err: <nil> Duration: 6.141535432s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 8.373576741s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 15:01:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.373603533s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................XXX...........XX.X................] duration: 19.033278711s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 15:01:39 UTC) orbiter warning: context deadline exceeded Duration: 19.033303747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X........X.........X......X.......X...X..] duration: 42.418183875s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 15:01:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 42.418210487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX...........] duration: 17.831339875s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:01:23 UTC) orbiter warning: context deadline exceeded Duration: 17.831365444s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X] duration: 30.003665921s trace: https://jaeger.dev.storj.io/trace/578f42818ce4b775 FAIL (2025-12-29 15:01:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA) context deadline exceeded ======LOGS====== Dec 29 15:00:46.230687400 (575h31m46.101770435s) [upload=504524]: Starting upload Dec 29 15:00:46.230745841 (575h31m46.101828886s) [upload=504524]: Done starting upload Dec 29 15:00:46.230852825 (575h31m46.10193586s) [upload=504524]: Uploading segments... Dec 29 15:00:46.230934129 (575h31m46.102017175s) [upload=504524]: Got next segment. Inline: false Dec 29 15:00:46.230993082 (575h31m46.102076117s) [upload=504524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 15:00:46.230999965 (575h31m46.10208301s) [upload=504524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 15:00:46.479113660 (575h31m46.350196705s) [seg_pos={0 0}, upload=504524]: Begin upload segment... Dec 29 15:00:46.479195305 (575h31m46.35027834s) [seg_pos={0 0}, upload=504524]: Done begin upload segment. Dec 29 15:00:46.479249448 (575h31m46.350332493s) [noise=true , address=45.91.116.9:4261 , piece_id=H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.479398883 (575h31m46.350481928s) [noise=true , address=77.90.31.12:4589 , piece_id=QPMAO6MQLJTOIUUGWYXHN7O6T3N6CKYMBMMK4LNTRCNI3GGKOW5A, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480061124 (575h31m46.351144169s) [noise=true , address=45.91.116.6:4976 , piece_id=6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480658030 (575h31m46.351741075s) [noise=true , address=45.91.116.7:4010 , piece_id=KMCWLJVCSTOZVIXSAM55C37VUATH6FW4O4HU7GUBQ6XSTBB56A2A, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480835388 (575h31m46.351918423s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.480968511 (575h31m46.352051556s) [noise=true , address=145.79.227.37:4567 , piece_id=LP5CCC2EJ32EWOJS272O2NX6DRQEHF7MXRUZY5IE5NCGNM6JFKJA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481156961 (575h31m46.352239996s) [noise=true , address=145.79.227.5:4223 , piece_id=6FM54KX3AAHCWBMSEKXNDXEWKR5F6BPXGLSVZLRX56T2TZ7M3LWA, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481228026 (575h31m46.352311061s) [noise=true , address=145.79.227.24:4582 , piece_id=KLSCJDC3CKBTMNIJLDPUMX5MGYOBFE2LXVHF2NLUCZKILTOY76IQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481611737 (575h31m46.352694772s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.481679526 (575h31m46.352762571s) [noise=true , address=77.90.31.10:4520 , piece_id=SP64YJGCQHZ7ODA5RLZMIFFQKZ54Z6WBTVEHN5INI6LJXZSTRSJQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482031055 (575h31m46.3531141s) [noise=true , address=45.91.116.14:4190 , piece_id=QDCAIZRYZNHDY3FZTWFELYLCRYHH2ZG2H2HNVPK4N5OSP2B5LYXQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482060171 (575h31m46.353143206s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482417301 (575h31m46.353500336s) [noise=true , address=145.79.227.8:4331 , piece_id=4Y352FONHTEJ7D2NX6FPJV2I3643LK4JL7HAEVVGJTTYQADGCHGQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482439292 (575h31m46.353522327s) [noise=true , address=145.79.227.27:4690 , piece_id=S54XALCWQ7MXEH7H36EWJZ7KJPCY4E7VYVNFTWDYYLVOBJZS7DSQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.482758561 (575h31m46.353841616s) [noise=true , address=145.79.227.15:4876 , piece_id=XSFNXDXL7WF45IQVNKI6Z6HOA643JZUOXFRKH6MUEVVQX4TWTRRQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:46.483191895 (575h31m46.35427493s) [noise=true , address=45.91.116.13:4154 , piece_id=E6IGXDVCKQJ6REFODY4R3CWVVLXC6MKRBPXIOEIXVNN7ACUF44BQ, seg_pos={0 0}, upload=504524]: Uploading piece... Dec 29 15:00:48.094503725 (575h31m47.96558677s) [upload=504524]: Next returned nil segment Dec 29 15:00:48.094520918 (575h31m47.965603963s) [upload=504524]: Waiting for error group managing segments... Dec 29 15:00:48.473958274 (575h31m48.345041309s) [noise=true , address=145.79.227.5:4223 , piece_id=6FM54KX3AAHCWBMSEKXNDXEWKR5F6BPXGLSVZLRX56T2TZ7M3LWA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.476468684 (575h31m48.347551729s) [noise=true , address=145.79.227.37:4567 , piece_id=LP5CCC2EJ32EWOJS272O2NX6DRQEHF7MXRUZY5IE5NCGNM6JFKJA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.483300322 (575h31m48.354383357s) [noise=true , address=145.79.227.8:4331 , piece_id=4Y352FONHTEJ7D2NX6FPJV2I3643LK4JL7HAEVVGJTTYQADGCHGQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.484038157 (575h31m48.355121192s) [noise=true , address=145.79.227.27:4690 , piece_id=S54XALCWQ7MXEH7H36EWJZ7KJPCY4E7VYVNFTWDYYLVOBJZS7DSQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.487580062 (575h31m48.358663097s) [noise=true , address=145.79.227.24:4582 , piece_id=KLSCJDC3CKBTMNIJLDPUMX5MGYOBFE2LXVHF2NLUCZKILTOY76IQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.490870848 (575h31m48.361953883s) [noise=true , address=145.79.227.15:4876 , piece_id=XSFNXDXL7WF45IQVNKI6Z6HOA643JZUOXFRKH6MUEVVQX4TWTRRQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.698180386 (575h31m48.569263421s) [noise=true , address=45.91.116.14:4190 , piece_id=QDCAIZRYZNHDY3FZTWFELYLCRYHH2ZG2H2HNVPK4N5OSP2B5LYXQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.707969877 (575h31m48.579052922s) [noise=true , address=45.91.116.6:4976 , piece_id=6LGX4VWCXQCII3GYTR4RUNHC3E7UBLHFK3NMGM2WJNOKGQEPQJGA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.749966284 (575h31m48.621049319s) [noise=true , address=45.91.116.7:4010 , piece_id=KMCWLJVCSTOZVIXSAM55C37VUATH6FW4O4HU7GUBQ6XSTBB56A2A, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.765112295 (575h31m48.63619533s) [noise=true , address=77.90.31.10:4520 , piece_id=SP64YJGCQHZ7ODA5RLZMIFFQKZ54Z6WBTVEHN5INI6LJXZSTRSJQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:48.814885480 (575h31m48.685968525s) [noise=true , address=45.91.116.9:4261 , piece_id=H5APLIWUC5A2XGQYY5K5YSCLJ4OWL7PDVWFC6QHPFZU2MTKBITHA, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.496478315 (575h31m49.36756136s) [noise=true , address=77.90.31.12:4589 , piece_id=QPMAO6MQLJTOIUUGWYXHN7O6T3N6CKYMBMMK4LNTRCNI3GGKOW5A, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.609434281 (575h31m49.480517336s) [noise=true , address=45.91.116.13:4154 , piece_id=E6IGXDVCKQJ6REFODY4R3CWVVLXC6MKRBPXIOEIXVNN7ACUF44BQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: <nil> Dec 29 15:00:49.609461562 (575h31m49.480544607s) [seg_pos={0 0}, upload=504524]: Segment reached optimal threshold of 13 pieces. Dec 29 15:00:49.609704565 (575h31m49.48078763s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.609714214 (575h31m49.480797259s) [noise=true , address=145.79.227.33:4869 , piece_id=SREQBVGHQ6UCQZ3X56VHIEA3WJJAPZGKONMAOGSBWXPHEL25U5IQ, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.609731626 (575h31m49.480814702s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.609743099 (575h31m49.480826164s) [noise=true , address=145.79.227.49:4338 , piece_id=TKDTI6WAT6HPYHKRNIWI4IC4D7VXD2GU2TXNRXU3IKC42YWXS24Q, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.610040595 (575h31m49.48112365s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 15:00:49.610050223 (575h31m49.481133268s) [noise=true , address=45.91.116.8:4048 , piece_id=U3WGKF5HDJ2VWI2K2ZQRARBTG4L6YMM347RRTRSQUAGZXSQR6MKQ, seg_pos={0 0}, upload=504524]: Long tail cancellation due to optimal threshold Dec 29 15:00:49.610076473 (575h31m49.481159508s) [seg_pos={0 0}, upload=504524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 15:00:49.610189077 (575h31m49.481272122s) [upload=504524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 15:00:49.610197915 (575h31m49.48128096s) [upload=504524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 15:00:49.906765905 (575h31m49.77784894s) [upload=504524]: Done uploading segments... Dec 29 15:00:49.906785301 (575h31m49.777868337s) [upload=504524]: Upload finished. err: <nil> Duration: 30.003691127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.002015797s trace: https://jaeger.dev.storj.io/trace/19114376637f8106 FAIL (2025-12-29 15:01:03 UTC) orbiter warning: context deadline exceeded Duration: 5.002040135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........................XXX...........XX.X...............] duration: 18.723651423s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 15:01:00 UTC) orbiter warning: context deadline exceeded Duration: 18.723680819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X........X.........X......X.......X...X.] duration: 47.219620428s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 15:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 47.219634655s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX..........] duration: 16.048084626s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:00:44 UTC) orbiter warning: context deadline exceeded Duration: 16.048130866s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX...........XX.X..............] duration: 19.581461906s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 15:00:35 UTC) orbiter warning: context deadline exceeded Duration: 19.581489188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 11.825415441s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 15:00:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.825442101s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.........] duration: 24.370552386s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:00:15 UTC) orbiter warning: context deadline exceeded Duration: 24.370585289s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 27.660904923s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:59:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 27.660928278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X........X.........X......X.......X...X] duration: 1m0.002613848s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 14:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 1m0.002623727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.001288875s trace: https://jaeger.dev.storj.io/trace/2200bb03b1d7bba7 FAIL (2025-12-29 14:59:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001312219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....] duration: 7.821117076s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 7.8211457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX...........XX.X.............] duration: 28.15765317s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:59:31 UTC) orbiter warning: context deadline exceeded Duration: 28.157670232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX....] duration: 5.969443514s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:59:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 5.969469444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX...........XX.X............] duration: 29.762936568s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:59:03 UTC) orbiter warning: context deadline exceeded Duration: 29.763098135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX........] duration: 16.898307171s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:59:02 UTC) orbiter warning: context deadline exceeded Duration: 16.898318494s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X........X.........X......X.......X...] duration: 49.581871703s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 49.581894195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 13.653840166s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:58:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 13.653867937s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.......] duration: 18.944169068s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:58:45 UTC) orbiter warning: context deadline exceeded Duration: 18.94419686s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.00051666s trace: https://jaeger.dev.storj.io/trace/40732a9f940e0851 FAIL (2025-12-29 14:58:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000540136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX...] duration: 5.80207739s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:58:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 5.802100252s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX......] duration: 23.183526034s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:58:13 UTC) orbiter warning: context deadline exceeded Duration: 23.183589014s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.002531239s trace: https://jaeger.dev.storj.io/trace/725eba0b4c5c0947 FAIL (2025-12-29 14:58:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002557781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X........X.........X......X.......X..] duration: 37.642485232s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:58:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 37.642506252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.487342543s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:57:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.487363812s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X..............................XXX...........XX.X...........] duration: 25.291424961s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:57:51 UTC) orbiter warning: context deadline exceeded Duration: 25.291474815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.....] duration: 22.420518139s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:57:50 UTC) orbiter warning: context deadline exceeded Duration: 22.420565109s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX..] duration: 6.077788297s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 6.077815279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......................X........X.........X......X.......X.] duration: 50.484019418s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 50.48403583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 15.305269207s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:57:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.305292311s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X..............................XXX...........XX.X..........] duration: 5.80351074s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:57:09 UTC) orbiter warning: context deadline exceeded Duration: 5.803537712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001812162s trace: https://jaeger.dev.storj.io/trace/7089f074b8ef247e FAIL (2025-12-29 14:57:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00184226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX....] duration: 20.625218087s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:56:58 UTC) orbiter warning: context deadline exceeded Duration: 20.625242825s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.......................X........X.........X......X.......X] duration: 1m0.002366233s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:56:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 1m0.002376602s core uplink us-select-1 us1-placement-20

1
...
11283
11284
11285
11286
11287
...
18081